Ok, I did some testing with an Adaptive Binary Tree in my application. I wanted to test if our current not fully optimized octree implementation (but still very very fast) is able to keep up with a ABT. I used the article in Game Programming gems 5 as my main resource. I am not going to explain what an ABT is because you can find that by googling. I must note that we are currently only using these structures for static geometry. An ABT is especially useful for dynamic objects but I think a loose octree also has potential. For now however I limit myself to static geometry only.
I am currently constructing the ABT in maya using the all vertices approach. This approach will result in best ABT but it also is a very slow method. However for the sake of testing whether an ABT is usefull for our application I decided to use this method. After a couple of hours I was able to render the adaptive binary tree with the same result as the octree implementation. I did not implement all the stuff the octree can do but it was enough for testing purposes.
I benchmarked the application and compared the result with the octree implementation. Traversal of the ABT was significantly faster than the octree implementation because there are fewer nodes. I compiled the ABT this way on purpose. Rendering however was much slower even though the rendering code of the ABT is much simpler. I noticed that the ABT draws a lot of triangles that are not visible. This was pretty obvious, the ABT has fewer nodes so the nodes must contain more geometry. But when I increased the number of nodes of the ABT the traversal time increased.
After a lot of testing I concluded that the ABT implementation is in fact slower than the octree implementation. I think it has to do with the fact that our levels do not contain a lot of geometry. At the current time our engine simply does not require an ABT.
So I think I will go and combat the fragmentation of octree nodes now to see if I can maybe squeeze frametime there.