Messaging and Directory Server software used around the world in the Government, Military, Aviation and Commercial sectors.

Whitepapers

Benchmarking M-Box - Isode's IMAP/POP Message Store

16th August 2005

Objectives

Isode's new IMAP/POP server, M-Box, was written with very high performance and scalability targets in mind. In order to judge our success in meeting these targets the product has and continues to be tested against a wide basket of commercial and open source alternatives.

This whitepaper gives performance numbers of M-Box using the Mstone industry standard benchmark and provides reference comparison with a number of widely used IMAP servers. This paper looks at single server performance ("vertical scaling").

We have used a widely available industry standard benchmarking tool and published our test configuration in order to encourage evaluators to conduct similar tests themselves. As with all Isode products we actively encourage evaluations of M-Box and will make support resources available to assist your evaluation.

Benchmarking: Why we used Mstone

Mstone is an industry standard stress and performance measurement tool, it runs on Linux, Solaris, AIX, OSF, HPUX, and NT, is free and available here. Our choice of Mstone, apart from the comprehensive nature of its reporting, was governed by our desire to make these tests repeatable by evaluators with no additional investment. It also provides vendor independent test selection, making it clear that we have not selected tests oriented to M-Box's capabilities.

Benchmarking: Our reference choices

We have chosen to use a number of popular Open Source products as references for M-Box performance. There are a number of reasons for this:

• The implementations are generally available and tests can easily be repeated by others.
• It is straightforward to test all products in exactly the same environment ensuring a level playing field.
• IMAP is popular in the Open Source community and these are considered to be good and stable implementations.
• Evaluation licences for commercial products prohibit the publication of test results.

The following implementations were chosen:

• Cyrus: though not renowned for it's horizontal scalability, Cyrus is recognised high-end open source product used as the base for a number of commercial implementations
• Courier: Popular with service providers, Courier provides horizontal scaling using a shared filestore and allowing multiple servers to be added 'transparently'. This approach, giving a substantial administrative benefit, is also used by M-Box, making it a good comparison product.
• UWash: Another popular IMAP server, UWash, like Cyrus, is not renowned for its ability to scale horizontally but is widely viewed as being very stable.
• Dovecot: A relatively new implementation, gaining in popularity.

Both UWash and Cyrus are considered to be IMAP reference implementations, they have relatively strict adherence to the IMAP protocols and are often used by authors of IMAP client software to test against.

Test System and Configuration choices

All products were built on the same server using the same compiler. Default install choices were selected for all products. The test system specifications were:

• Operating System: Gentoo Linux 2.6.13
• Memory: 1 Gig
• CPU: Pentium(R) D CPU 2.80GHz GenuineIntel (Dual Core) CPU
• Disk: WD Raptor SATA 10K
• Compiler: gcc 3.3.5
• Thread model: posix
• File-system: reiserfs

The Results

The tests simulated multiple clients accessing each of the servers (for these tests we chose simulations of 100, 500 and 1,000 clients) and recorded all performance measures relevant to the access.

M-Box performed excellently across the board as you'll see from following the links to the detailed test results below each set of graphs. For the purposes of this paper we've presented summary results that illustrate the number of transactions attempted over the 5 minute period of each test, the number of errors generated by those transactions and Gigabytes read during the test period.

 

Follow the links for the full results at 100 simulated clients for Dovecot, UWash, Courier, Cyrus and M-Box.

As expected all products performed reasonably well when subjected to low (100) simulated client loads, although there is a a significant difference in the number of attempted transactions even at this level with Dovcot performing particularly poorly.

Many of these transactions could just be simple searches, however the 'Gigabytes Read' graph shows that M-Box is doing more 'real' processing work than any of the other systems, with only Cyrus approaching it's performance.

 

Follow the links for the full results at 500 simulated clients for Dovecot, UWash, Courier, Cyrus and M-Box.

As the simulated client load increased to 500, the performance differences between M-Box and the other systems under examination became more pronounced, particularly with regards to errors. All the errors result from a read, write, or connect timeout of 60 seconds. In practice this means that the client (user) was unable to login and was denied service.

M-Box shows only a moderate reduction in overall transaction rate from 100 connections and operates comfortably with this level of connections. The level of errors with the other IMAP servers suggests that they are unable to provide a satisfactory service at this load level.

 

Follow the links for the full results at 1000 simulated clients for Dovecot, UWash, Courier, Cyrus and M-Box.

The figure for errors become particularly pronounced at 1000 simulated clients with M-Box maintaining a zero error level across all simulated client loads.

The other IMAP servers are refusing a significant percentage of connections at this load and are well beyond viable operational load. We estimate that at this loading, Cyrus was only providing a service to about 600 of the 1,000 clients. Whilst M-Box's overall transaction performance dropped between 500 and 1,000 active connections it was still operating at a more than satisfactory level for a server of this size and the Gigbytes Read chart shows that it was handling substantially more real processing work thn any of the other systems. Of particular note is that it supported all of its connections without error.

Inbound Message Performance

The figures shown so far relate to client access to the IMAP Server (getting messages out). For overall performance, getting messages into the IMAP Server is also important.

Some IMAP Servers have messages written to a mailbox format by the associated Message Switch. Courier, Dovecot and UWash servers all work in this manner. Delivery performance for these servers is primarily a function of the associated Message Switch (generally, there is a choice of Message Switch that can be used with the IMAP server) so this measurement is not appropriate for this paper, which is considering the IMAP server.

Cyrus and M-Box support direct delivery, using LMTP (Local Message Transport Protocol). Delivery performance was measured for 1,000 4 kbyte messages, for these two servers.

The significantly lower delivery time for M-Box illustrates one of the advantages of using a relatively simple file store aligned mailbox format.

Conclusions

These tests have illustrated the scalability of M-Box compared with popular IMAP alternatives. The performance of M-Box is consistent over all the loads levels tested, efficiently sharing available resources amongst all the active users.

M-Box performs particularly well with regards to errors with no timeouts at any test load.