This paper provides Benchmark information for Isode’s M-Vault R15.1 directory server.
The basic test setup is simply a test client, connecting to an M-Vault server. We used Isode developed test clients, because:
- In order to obtain maximum read/search performance, significant care is needed in how the client works.
- We need a test tool to run efficiently on a system that is somewhat less powerful than the server under test. Some open source tools would need much faster machines to generate sufficient load.
The server used for tests is:
- Model : HP Proliant DL585
- CPU : 4 x Quad-Core AMD Opteron Processor 8356 (2.3GHz)
- Disk : HP 146GB 10K SAS 2.5" Single Port
- Memory: 64 GByte
- OS: Centos 5.5
This is a fairly fast server machine, about 4 years old. It is a reasonable server to do benchmarking on, although a new machine would be significantly faster.
For the tests, the M-Vault server was configured as follows:
- Unlimited entry cache. We would recommend this for a large deployment on appropriate hardware.
- Configured with equality indexes on 'surname' and 'mail'. This is appropriate for a simple directory setup, In a complex directory deployment, more indexes will be needed. This would impact performance, in particular write performance.
- Default access controls in all cases (except where noted). Access control checks can have significant impact on directory performance. The Isode default access control is appropriate for many types of directory service.
- Audit logging disabled (except where noted). Audit logging has a significant effect on directory performance. Most tests were done without audit logging, as this gives a useful measure of directory performance. Measurements are also given to show the impact of audit logging.
The tests were conducted on directory servers containing 1 million and 10 million entries in a flat Directory Information Tree. The data is systematically generated. Example entry:
dn: cn=Narida Valcourt-Zywiel,o=Test,c=BA
cn: Narida Valcourt-Zywiel
Read and Write Performance
Read tests were done with a test client that maintains 64 open connections to the directory, and sends multiple (asynchronous) queries down each connection. Queries are made randomly across all of the entries. The client operates so that it provides maximum load on the server, but with load flow controlled to prevent over-loading the server.
Modification tests modify the value of one attribute within an entry.
M-Vault will cache results as it operates, which means that it gets faster as the cache builds. Eventually, a stable maximum performance value is reached, and it is this figures that is measured. Initial read performance is less than this stable value, as information is not initially cached in memory. An operational DSA will be a long running process, so the stable value seems the most useful measurement.
Search Rate (operations/second) over time (seconds) for 1 million entries
|1 Million Entries||10 Million Entries|
|Memory Pool Used||670 MByte
|Process Resident Set Size||2.3 GByte||30 GByte|
|Time to reach full speed||2.5 minutes||34.5 minutes|
|Test||1 Million Entries (ops/sec)||10 Million Entries (ops/sec)|
|Max search rate (no attributes returned)
|Max search rate (all attributes returned)||103,000||98,000|
|Max search rate (no access control, no attrs)||110,000||105,000|
|Max search rate (w/audit logging, no attrs)||24,000||18,000|
|Max modify rate||5,000||4,300|
|Concurrent search and modify||85,000 (search)
A number of observations can be made on these numbers:
M-Vault can support over 100,000 searches per second, with no significant performance degradation at 10 million entries. Search rate with a very small database is not significantly faster.
Returning a small number of attributes has minimal performance impact relative to returning no attributes.
The M-Vault default access control setup does not impose significant overhead. Note that complex access control could have much greater performance impact. The cost of sensible basic access control is low, and so there is no reason to run a server without access control.
There is some degradation in write performance in moving from 1 million to 10 million entries, but the change seems reasonable.
For a server doing active reads and writes, the read performance has moderate degradation from the maximum value. This is very good, as it shows that read performance is not significantly impacted by directory updates. Modification performance drops to just under half, which seems reasonable.
Audit logging has a massive impact on search performance. When audit logging is turned on, performance is limited by audit logging. It is clear that where audit logging of read access is required for a high volume server, careful consideration should be made of the audit logging configuration (e.g., fast disks for audit logging, separate from the disks used for the database).
The search data above gives useful information for applications where the directory client can keep long lived connections to the directory. A common directory use case is where a client binds, does a lookup and then closes the connection, often without authentication. Also the directory is often used for user authentication, in order provide an authentication service for another application. The following measurements were made on a directory with one million entries:
|Simple Bind, re-using LDAP connection||60,000|
|Simple Bind, with new LDAP connection for each bind||8,000|
|SASL with CRAM-MD5 re-using LDAP connection||17,000|
|SASL with SCRAM, re-using LDAP connection||400|
The first two tests use simple bind, which is a common approach, particularly when using the directory as an authentication server for other applications. Where existing LDAP connections are used, simple authentication is very fast.
Where a connection is established for each bind, the rate is much slower and essentially is limited by the cost of establishing a new TCP connection. If TLS was also used, this would increase the per connection cost significantly.
CRAM-MD5 is an authentication mechanism that protects the password exchange in protocol. This will use a plain password stored in the directory.
SCRAM is a new authentication mechanism that is expected to gain increased use. It provides both protection on the wire and a hashed format stored in the directory. SCRAM uses a hash mechanism applied a configurable (and typically large) number of times to slow down checking. This is to prevent brute force analysis, particularly in the event of capture of information from the database. So the slow performance here is a consequence of this security.
Adding Entries to the Directory
Performance for adding data to the directory is important. Using LDAP directory adds over protocol, 1 million entries were loaded in 23 minutes (725 per sec).
As a bootstrap operation, it is often necessary to load data into the directory form an external source. M-Vault provides an optimized “direct to disk” loading tool. This is able to load 1 million entries in 3 minutes (5,500 per second).
When a large directory service is deployed, it will invariably use multiple servers to provide high availability, with replication between the servers. M-Vault provides this using the efficient X.500 DISP protocol. It is important that this configuration does not significantly impact the performance of individual servers.
A total update of 1,000,000 entries from supplier to consumer took 6.75 minutes.
A master server was modified at 1,600 entries per second, which was shadowed to a consumer. When there was no load on the consumer, update was sub-second. When the consumer was loaded with searches of 40,000 operations per second, update after an extended period completed in 4 seconds. This shows that replication can “keep up” under high load conditions.
When operating at the top end of M-Vault performance, network throughput is a significant consideration. The tests were all performed over GigaBit Ethernet. This has a theoretical throughput limit of 125 MBytes per second, which in practice is not achievable by any network application. For the tests run, some had a measured data rate of 40 MByte per second. Although this is not at the theoretical limit, it is likely that the limit on network performance had some impact on the results.
Very Large Databases
We made measurements on M-Vault with databases of 10 million entries. This was chosen as a large database size for which we believed the machine chosen had suitable capacity for a directory of that size. We believe that M-Vault is suitable for deployments of at least 100 million entries. For a deployment of that size, we would recommend the use of faster hardware with more memory.
Is M-Vault Fast Enough for Me?
A simple goal of this paper is to show that M-Vault is a very fast directory. For many deployments, M-Vault will be amply fast enough and no detailed consideration of performance will be necessary.
For deployments with high performance requirements, significant care needs to be taken. The measurements here are generic, and performance will be dependent on the nature of the exact loading. It is not practical for Isode to measure all of the combinations of load that are possible. Isode recommends making benchmark tests with load appropriate to the target environment. Isode is happy to help customers, prospects and partners with setting up such tests.
A common problem in high performance environments is that performance gets limited by the client rather than the server. This needs to be considered both when making performance tests, and as part of the deployment.
M-Vault configuration will be important for high performance configurations, including choice of indexes and setting cache sized.
Hardware choice is important for a high performance deployment. Considerations include amount of memory, and number and type of disks to be used.
Comparison with other Directory Servers
This section considers briefly how M-Vault R15.1 performance compares with other directory servers.
We previously benchmarked M-Vault in 2003. It can be seen that software and hardware improvements have led to at least a fifty-fold increase in performance.
Comparing R14.6 M-Vault on the same hardware as the tests above, shows that R15.1 has a six-fold increase in performance over the older directory server
We compared M-Vault performance to two open source LDAP servers:
- OpenLDAP is a well-known open source LDAP server, which was also used as a reference in 2003.
- OpenDJ is a new open source project forked from the Sun OpenDS project.
The following graphs show comparative performance. We worked to get best performance out of each server. The results suggest that M-Vault has the best Read Performance:
Read/Search Performance (operations per second)
The following table shows 'modify only' and 'search - modify' performance. We can see that M-Vault's modify only performance is lower than both OpenLDAP and OpenDJ.In a mixed search-modify environment, which generally reflects real conditions, M-Vault maintains its large lead in search performance with the relative modify rates also reflecting those reported in the modify only test.
|Modify only (ops/sec)||Search - Modify (ops/sec)|
The resident set size for M-Vault is significantly lower than OpenLDAP and a little lower than OpenDJ:
Resident Set Size (GB)
For the search results, OpenLDAP reached peak performance somewhat faster than M-Vault and OpenDJ was somewhat slower. OpenDJ performance had regular "downward spikes", which may be an operational issue in some deployments (we suspect caused by Java Garbage Collection).
To our knowledge, these two servers are the best comparisons. We have seen benchmarking numbers that we trust, which show OpenDJ and OpenLDAP performance, and a commercial directory server from a large vendor we cannot name at about 50% of OpenLDAP search performance.
We are not aware of any other commercial or open source directory servers that have comparable performance.
This paper gives performance numbers for M-Vault. They show that it is one of the fastest directory servers available.