In the mid 1990s I was working on a project to develop a large-scale user and access management system. Because this system had to be able to scale to large numbers of users, it seemed a good idea to use an X.500 directory to store the user data. We also liked the X.500 data model because it provided excellent syntactic control over entries. One of the directory servers we looked at was an X.500 directory server from ICL. While we were discussing the project goals with an ICL engineer, he stopped, cocked his head, and said “You aren’t actually going to populate the user interface with entries from the server as they’re returned, are you?” I replied that, yes, that’s exactly what we wanted to do. He proceeded to explain to me that that was a very bad idea because ICL’s server would only return between one and two entries per second and it would therefore take a very long time to populate the user display. This, it seemed, was the state of the art for directory servers; you could have lots and lots of entries in them, but they serviced search requests very slowly. The project was scrapped soon after and I moved on to other things, but I didn’t forget that moment when that very proper British engineer looked at me like I was barking mad because I expected his directory server to return search results quickly.
Fast-forward eighteen years, and I’m a principal at the company that develops and supports OpenLDAP, a directory server which has the distinction of being the fastest LDAP (a derivative of X.500 directories) directory server on the planet. Not only is it capable of returning hundreds of thousands of search results per second using garden-variety server hardware, it scales to billions of entries in a single server while maintaining a linear response profile. The speed, scalability, and linear response profile of the server invites uses we didn’t even dream about when I first came into contact with directories. At its core, OpenLDAP is simply an object-oriented database that’s organized hierarchically. It’s not column-oriented and it doesn’t do joins. It delivers on promises made in the early eighties for a database that could solve problems that are intractable with conventional row-and-column oriented database systems. Simpler data retrieval applications- most web applications, for example- no longer need to suffer the performance and complexity penalties imposed by SQL parsers. Instead, they can issue LDAP search requests that can be satisfied in 600 microseconds or less with modern commodity hardware.
This is not just a boast. There’s been a generational change in object-oriented database technology, and it’s happened right under our noses, almost without us noticing. Sure, we can now run many thousands of authentications per second, but it also suddenly becomes reasonable to run permission checks in real time instead of at application startup or login. Web service requests can be serviced more quickly and voicemail messages stored and retrieved more efficiently. MMO games can run more smoothly, without delays. It’s time to re-think our database technology choices and take advantage of the opportunities presented by this oft-ignored and underutilized technology.