Messaging and Directory Server software used around the world in the Government, Military, Aviation and Commercial sectors.

Whitepapers

Benchmarking M-Vault: Isode M-Vault vs. OpenLDAP

Publish Date: 16th October 2003

Purpose

This white paper describes LDAP (Lightweight Directory Access Protocol) performance benchmark tests of Isode's M-Vault directory server, and gives comparison benchmarks with OpenLDAP. Tests were performed using the independent DirectoryMark tests, on a small Linux server, with database sizes from 100,000 to 1,000,000 entries.

The first part of this paper gives an outline of the tests and the rationale behind them. The second part deals with the details of the tests and results. The tests showed that M-Vault is significantly faster than OpenLDAP.

Benchmarking Framework

Comparison Servers

Isode’s primary goal in this benchmarking is to provide information on the performance of M-Vault to its customers and prospects. To put this information in context, benchmarks were also done against OpenLDAP.

OpenLDAP is a popular open source LDAP implementation. It is a straightforward and simple implementation of LDAP, and thus provides a useful independent reference point.

Benchmarking Platform

The benchmarks were run on a dual Intel Pentium III (1.4 GHz) system with 1Gbyte memory running Red Hat Linux. This low cost platform was chosen because:

• Use of a small server will ensure that system limits are stressed.
• Use of small servers is becoming an increasingly popular way for organizations to deploy services.
• Organizations are often persuaded to run services on unnecessarily large (and expensive) hardware, and it is useful to show that it is quite viable to run a directory server on a relatively small system.

Benchmark Tests

The performance tests were run using the DirectoryMark test suite. This test suite was developed by Mindcraft, who are an independent company specializing in benchmarking and test systems.

Test Fairness

These tests were conducted by Isode staff (not by an independent third party). We worked hard to measure consistently, and to set parameters which gave best performance for each system. We would welcome and assist anyone attempting to independently verify these results.

We did not test on all the platforms supported by Isode (Solaris and Windows 2000, as well as Linux), and the comparative results could vary with operating system and hardware choice.

Analysis of Results

General

Detailed results for all the tests are given in the companion paper. The following table shows the fastest and slowest result for each product, across all of the tests and database sizes.

For all tests, Isode's M-Vault was faster than OpenLDAP, and in most cases considerably faster.

Memory and Disk Operation

The amount of memory on the system, meant that for the 100,000 entry tests is ample to enable all entries to be held in memory. For the1,000,000 entry test, there is insufficient for this to happen, and so these test cases clearly test different modes of operation of the server. For 500,000 entries, the servers were "on the edge", and results for this set of test showed the widest variance between test runs.

Read and Search

The "messaging" and "address" tests focused on read and search functions. The messaging tests use tests which match entries exactly and return a small amount of information. This pattern of usage is intended to simulate a messaging system's use of the directory. Isode and Sun had similar performance for these tests, with Sun faster in the majority of tests.

The address tests include a mix of searches and reads. Analysis of the logs suggests the performance results are dominated by the performance of substring searches (wildcard searches). The number of results for each of these searches is proportional to the size of the database, so performance is slowed both by the increase in size of database and by searches returning more results. This scenario is intended to simulate white pages or address book type usage.

The tests are conducted with a varying number of clients, to simulate different operational situations. Because the clients do not pause between operations, test scenarios would likely reflect an operational situation with a higher number of clients.

Writing and Loading

The "mixed" tests combine a variety of directory operations, to simulate a general purpose directory environment. Directory add and modify operations significantly determine the results of these tests. In all cases, Isode was faster than Sun in these tests.

We also measured adding 1,000,000 entries over protocol, which is a good measure of the directories performance when adding entries. Isode took 7 hours 7 minutes and Sun took 13 hours 36 minutes. OpenLDAP’s performance made this test impractical.

All of the products support a "direct to disk" loading mechanism, which is a fast way to load data, although it is not always operationally desirable or practical. For 1,000,000 entries, Isode took 2 hours 15minutes and OpenLDAP tool 20 hours. It would be operationally awkward to use OpenLDAP for this size of directory, particularly if replication was needed.

DirectoryMark Test Framework

Platform

The tests were run on a Dell Intel system with:

• 2 x 1.4 GHz PIII processors (512Kbytes L2 cache)
• 1 Gbyte memory
• Adaptec 2100 RAID controller (write back cache)
• 9 Gbyte SCSI disk (9 ms average seek time)
• RedHat Linux 7.3 (updated; 2.4.18-27.7.xmtp) kernel

Test Method

The main testing was done with the DirectoryMark V1.2 test suite. These tests were developed by Mindcraft, an independent company specializing in benchmarking and test systems. DirectoryMark provides a client (run on a Solaris system). Multiple client processes can be run, and each client can be multithreaded. In this way a single tester system can simulate many clients. Each client thread reads a pre-prepared script of LDAP operations. The test either runs through the script once, or proceeds for a given time, starting the script again if the end is reached.

The DirectoryMark perl scripts which generate the test scripts were modified to ensure that:

• Wildcard (substring) searches used sufficient characters, and spaces were not at the end of the substring. (The DirectoryMark tests only use leading substring matches. Some Directory Server products do not support other substring types).
• The correct attribute type is used for ‘unique identifier’ matches.
• The DirectoryMark software also provide scripts to generate data to be loaded. The resulting data generates values for the seeAlso attribute, which are invalid (the seeAlso attribute type has DN syntax). So these attribute values were removed prior to loading.

The same test scripts were used for all platforms. It is not believed that the test system was a limiting factor in the tests.

The clients were configured not to delay between operations. So the load reflects rather more clients in a real scenario, where each client would not be sending operations continuously to a server.

Test Scenarios

There are three basic test scenarios:

• Messaging. The client perform LDAP subtree Searches with an exact match filter which matches a single entry. This is designed to simulate the load a messaging environment might place on a directory.
• Addressing. The client performs a mixture of exact and substring Searches (leading substrings only) on a variety of attribute types. This is designed to simulate the load that a white pages types service might place on a directory.
• Mixed. The client performs a mixture of operations:

Configuration

As far as possible each server was configured in a similar fashion:

• The entry cache was set as 125000 entries
• The database cache was set at about 250Mbytes
• The look-through limit (the number of entries considered in a search) was set at 5000 (OpenLdap does not appear to make this configurable).
• The search limit (the maximum number of entries returned in a search) was set at 200.

The cache sizes were set to make the memory footprint of the server process less than the physical memory size of the system. All of the software appears to use the standard Linux heap management software. This has an issue in that with a multithreaded process (and all the systems under test are multithreaded) can experience uncontrolled memory growth.

Memory freed by one thread is not reused by another thread. This means that for all software being tested the final memory footprint was rather larger than predicted by a calculation of the expected use from the entry cache and other caches.

Databases

Tests were performed with databases containing 100,000, 500,000 and 1,000,000 entries. The middle case was interesting in that this was rather on the shoulder of the performance curve. Performance is sensitive to the exact size of database files and other factors.

Test methodology

For each server being tested and for each database size a test run performed the following steps:

1. The database is created
2. The database is loaded using ‘direct’ tools, and the indexes generated (if index generation is a separate step).
3. The server is started
4. The clients read each entry once. This loads the entry cache into a fixed state.
5. The messaging class tests are run in turn with increasing numbers of clients
6. The address class tests are run in turn.
7. The mixed class tests are run

DirectoryMark Results

In these results, the best values from different runs are recorded here. Particularly for the 500,000 entry case, there was significant variation.

The load time is expressed as hours:minutes:seconds. All other numbers are the average operation rate in operations per second, as reported by DirectoryMark.

100,000 Entry Database

500,000 Entry Database

1,000,000 Entry Database

Loading over Protocol

The main tests used the direct tools for loading databases, as this is always faster. However, partly to measure the performance of LDAP Add operations, and partly because direct loading is not always possible a load test was performed 'over protocol'. Direct loading requires that the main server is not running.

The load operation was done using 4 clients running on another system. The client was not the DirectoryMark client, but the Isode ltcldish client. The 1,000,000 case was loaded, and the time for each 200,000 entry tranche was noted.

This test with OpenLDAP was abandoned, as it was so slow, it would not have been practical to let it run to completion (It took 3.5 hours to load 60,000 entries. At the final load rate, it would have taken 79 hours to load 1,000,000 entries, although significant further slowdown appeared likely). Times are hours and minutes, and operations are adds per second averaged over the interval:

 

Conclusions

A common myth in the directory industry is that "pure LDAP" servers are orders of magnitude faster than directory products that support both LDAP and X.500, because "LDAP is fast and light, whereas X.500 is heavy and slow". While some X.500 products are much slower than the servers measured here, these numbers clearly show that a directory product can support X.500, with all its contingent benefits such as standardized access control and replication, and be fast.

These results also show that good performance results can be achieved for 1,000,000 entry directories, using quite modest hardware.