M-Box is a high performance Internet Message Store, supporting IMAP (Internet Message Access Protocol) and POP (Post Office Protocol). M-Box uses LDAP (Lightweight Directory Access Protocol) configuration and offers good horizontal scaling. M-Box is a key element in Isode's Mailbox Services and provides the message storage and access component of an Internet Messaging solution, as illustrated below:

M-Box as part of a Messaging Solution

Email clients can access messages in M-Box using either POP or IMAP. Message switching and related services such as spam filtering are handled by M-Switch, which delivers new messages into M-Box using LMTP (Local Message Transfer Protocol). M-Box gets configuration information, including client authentication from an LDAP directory, such as M-Vault.

A more detailed example showing M-Box in the context of a full configuration is given on the Mailbox Services page. Horizontal scaling (running M-Box on multiple servers) is described later in this page. You can evaluate all the elements of Isode's Internet Messaging Solution (including M-Vault, M-Switch and M-Vault).

Key Benefits

M-Box has many functions, which are described below. Reasons why this product may be of particular interest include:

  • High performance. M-Box has an optimized database design and multi-threaded architecture, which gives high performance and allows a very large number of users to be supported on a single server.
  • Horizontal scaling. Multiple M-Box can robustly share a single file store, enabling straightforward service scaling for very large deployments.
  • Simple database format. M-Box uses a simple format mapping directly to the underlying file store, with one message per file and one directory entry per account. This gives natural robustness, and simplifies account provisioning.
  • LDAP directory based configuration. M-Box stores configuration and authentication information in the Directory, which allows sharing between multiple instances of M-Box and M-Switch, as well as enabling straightforward integration with a provisioning system.
  • Web based management. The Internet Messaging Administrator allows email administrators to create and manage user email accounts and shared folders, edit the message store configuration (including message size quotas and delivery options) and examine the current directory configuration.
  • Mobile device support. M-Box is designed to support mobile email devices, and in particular has support for some of the LEMONADE specifications.
  • Rock Solid. M-Box has exceptional robustness and stability, including support for fail-over clustering and Off Site Hot Standby (Disaster Recovery).
  • Online backup can be performed using standard backup tools, as a consequence of the M-Box database architecture. This enables straightforward disaster recovery.
  • Security. There are a variety of security features, including SASL for POP and IMAP access.
  • Audit and event logging will be added in a future release, using the same mechanisms as other Isode products. The current release uses syslog for critical events.
  • Flexible message processing on delivery, controlled by SIEVE.

M-Box Architecture

M-Box Architecture

An M-Box server is based around a message database, and has three multi-threaded servers that access this database:

  • imapd: supports access using IMAP.
  • pop3d: supports access using POP.
  • lmptd: supports message delivery using LMTP.

The most important part of the M-Box architecture is the database design. M-Box uses a database abstraction layer, so that it is possible to use different database back-ends. This has been prototyped with several back-ends, although a single database back-end is currently being provided. Future releases of M-Box will take advantage of this architectural flexibility.

The underlying database is mapped onto the underlying file store using a format which we call “mdir”. This uses the basic approach of the popular maildir format, which enables mailbox sharing without use of file system locks. Mdir differs from maildir primarily to enable performance improvements, particularly in support of IMAP. Key features of the mdir format:

  • No locks. This is critical for high performance, and robust horizontal scaling.
  • Each message in a single file. This simple approach gives inherent high robustness, as any data corruption will be constrained to a single message. It also allows standard file system backup mechanisms to be used for online backup.
  • Use of file store hierarchy. Folders and mailboxes are structured using the file store hierarchy. This approach is inherently robust and allows use of standard OS tools on the database. A key benefit is that each account is represented by a single directory hierarchy, which simplifies provisioning support and in particular account deletion.

A quick analysis might suggest that this simple approach, while very robust, would not give good performance. In fact, exactly the opposite is true. Modern files systems are highly optimized for exactly the sort of operations that are performed by M-Box. In particular:

  • File system directories are indexed using b-trees, and have optimized layout on disk. Use of large directories with key indexing information in the filename is therefore very efficient.
  • Files are intelligently laid out on the disk, which means that use of relatively small block size (good to gain high disk utilization, as messages are generally small) does not impact performance for large files.

In essence, M-Box is working with the tuning of the underlying file system, which is one of the reasons for its excellent performance. Our tests have shown that M-Box is faster than servers that use complex approaches to optimize performance, and that the approach taken gives benefits of simplicity, robustness and high performance.

Horizontal scaling

A key benefit that mdir provides is support of horizontal scaling as illustrated above. This architecture allows multiple M-Box servers to access a single file store. This architecture gives 'horizontal scaling', and enables additional M-Box servers to be added to support more users. Load balancing between the M-Box servers will usually be achieved by use of a network load balancer. The file store will be chosen with appropriate size, performance and expansion options for the deployment. It may be a file system on one of the M-Box servers, but in most cases it is likely to be a separate network file server appliance. The file access protocol will depend on performance and scaling targets for the total system. Options include:

  • NFS (Network File System). The traditional protocol for this sort of functionality.
  • iSCSI. A mapping of the SCSI (Small Computer Systems Interface) protocol commonly used onto IP. iSCSI can be used with a clustering file system to provide a low cost, higher performance alternative to NFS.
  • SAN (Storage Area Network). A more expensive, higher performance option.

Although this approach gives an elegant way to scale an M-Box deployment by adding more servers, the major advantage of this architecture is in simplified management. This architecture leads to user mailboxes being deployed on a single file store, with each account having a separate directory. Increase in storage is simply a question of increasing the size of the file server. This makes it easy to add and remove accounts.

Many mailbox servers, particularly those with complex databases, lock mailboxes to specific servers. This leads to two problems:

  1. In order to make mailbox location transparent to the end user, a tier of application servers will generally be introduced, so that the user gets redirected to the right server.
  2. When servers are added, or usage grows, there will be the requirement to migrate users between servers. This adds product and operational complexity.

M-Box's architecture avoids these problems.

Performance

M-Box is fast. Our own benchmarking indicates that M-Box is significantly faster than open source implementations and commercial competitors. Whilst license restrictions prevent us from publishing comparison figures against commercial competitors, we'd encourage you to perform your own tests and will support you in those tests.

While there is much detail to consider, there are three basic questions to answer with regards to performance:

  1. How fast is the database? This is best tested by a single connection performing operations on a mailbox. M-Box performs much better than most, and as well as the best (Cyrus).
  2. What changes when you add multiple connections? M-Box's multi-threaded architecture means that it performs significantly better than the other stores.
  3. Does it scale horizontally? The other stores that offer horizontal scaling have very bad answers to Q1 and Q2.

IMAP and POP Protocol Support

IMAP and POP are both protocols with a large number of optional extensions. M-Box has comprehensive support of these optional extensions, making it ideal for environments supporting a wide mix of email clients. Details of protocol support are listed in theInternet Mail Conformance section below.

M-Box is designed to support mobile devices, and conformance to some of the new LEMONADE specifications is included, prior to final standardization of this work. The IETF's LEMONADE (License to Enhanced Mobile Oriented And Diverse Endpoints) working group is addressing messaging standard extensions for optimized open standard mobile messaging. You can read more about LEMONADE and Mobile Messaging in our whitepaper, [Open Standards: The best approach for mobile messaging].

SIEVE Processing

SIEVE, which is a delivery time scripting language for email processing is supported by lmtpd. SIEVE enables users to manage actions such as automatic email filing, and vacation alerts. Personal sieve scripts are configured in the directory and may be managed by IMA. There is also a "global" sieve script, so that custom processing can be defined for all users.

Configuration Management

The configuration management of M-Box is integrated with the directory based configuration of the messaging infrastructure. M-Box configuration information is held in the (LDAP) directory, which enables client/server management of data and sharing of information between M-Box servers. This includes configuration information for the M-Box server, but the most important information is on users.

The core model is "one user; one account; one directory entry". While this can be managed with Isode’s directory data entry tools, in many deployments account management and user option selection will be integrated with the customer provisioning system, with the directory forming a back end to this.

More details, including the LDAP attributes used to configure M-Box, can be found in the M-Box Administration Manual, available for download from this site. M-Box server configuration is done using the Web based Internet Messaging Administration (IMA) applications.

Account Administration

Web-based administration if provided by the Internet Messaging Administration application.

Email administrators can use the IMA to:

  • Create and manage user email accounts
  • Create and manage shared email folders
  • Create and manage groups of users (used in access control lists and for shared folders)
  • Edit the message store configuration (including message size quotas and delivery options)
  • Examine the current directory configuration.

You can find out more about the Internet Messaging Administrator by clicking here.

Operational Management

M-Box specific operational control is provided by the msadm tool, which enables checking of accounts and msstat which provides connection information. Msadm provides scripted creation of accounts, and enables message expiration, so that old message can be removed. M-Box provides audit logging of key events, and uses Isode events for errors and operational information. This is described in the page on Event Handling and Audit Logging.

M-Box provides SNMP monitoring of general status and network connection. Events may also be monitored by SNMP, with alerts using SNMP Notifications (TRAPs). Further information is given on the page Isode's SNMP Architecture.

Migration

M-Box provides a fully automated, on-the-fly migration system, capable of migrating both message data and authentication information from an existing POP or IMAP service.

In a one-shot variant of the Gateway model, the single POP mailbox, or all the user's personal IMAP mailboxes, are migrated dynamically on login, using only standard, secured POP3 or IMAP connections. On completion, the user is automatically provisioned with an account in the directory.

In the majority of cases, this allows for a switchover from an existing system entirely transparent to the user, with no pre-population of account data required.

This reduces both the downtime and administrative load dramatically, allowing rollout to be achieved with zero-downtime. 

Authentication & Security

Authentication of IMAP and POP users may be performed using SASL, built over Isode's SASL infrastructure. Data confidentiality may be provided either by SASL and/or by TLS.

M-Box supports IMAP Access Control, which can be managed by IMAP clients, for example to enable folder sharing,

Internet Mail Conformance

RFC 1123 Requirements for Internet hosts - application and support. R. Braden, October 1989
RFC 1734 POP3 Authentication command. J. Myers, December 1994
RFC 1939 Post Office Protocol - Version 3. J. Myers, M. Rose, May 1996
RFC 2033 Local Mail Transfer Protocol. J. Meyers, October 1996
RFC 2034 SMTP Service Extension for Returning Enhanced Error Codes. N. Freed, October1996
RFC 2045 Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies. N. Freed, N. Borenstein, November 1996
RFC 2046 Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types. N. Freed, N. Borenstein, November 1996
RFC 2047 Multipurpose Internet Mail Extensions (MIME) Part Three: Header Extensions for Non-ASCII Text. K. Moore, November 1996
RFC 2049 Multipurpose Internet Mail Extensions (MIME) Part Five: Conformance Criteria and Examples. N. Freed, N. Borenstein, November 1996
RFC 2087 IMAP4 QUOTA extension. J. Myers, January 1997
RFC 2088 IMAP4 non-synchronizing literals. J. Myers, January 1997
RFC 2177 IMAP4 IDLE command, B. Leiba, June 1997
RFC 2342 IMAP4 Namespace. M. Gahrns & C. Newman, May 1998
RFC 2449 POP3 Extension Mechanism. R. Gellens, C. Newman, L. Lundblade, November 1998
RFC 2595 Using TLS with IMAP, POP3 and ACAP. C. Newman, June 1999
RFC 2920 SMTP Service Extension for Command Pipelining. N. Freed, September 2000
RFC 3462 The Multipart/Report Content Type for the Reporting of Mail System Administrative Messages. G. Vaudreuil, January 2003
RFC 3501 Internet Message Access Protocol - Version 4rev1. M. Crispen, March 2003
RFC 3516 IMAP4 Binary Content Extension,L. Nerenberg, April 2003
RFC 3691 Internet Message Access Protocol (IMAP) UNSELECT command. A. Melnikov, February 2004
RFC 3894 Sieve Extension: Copying Without Side Effects. J. Degener, October 2004
RFC 3848 ESMTP and LMTP Transmission Types Registration. C. Newman, July 2004
RFC 4314 IMAP4 ACL extension. A. Melnikov, December 2005
RFC 4315 INTERNET MESSAGE ACCESS PROTOCOL - UIDPLUS extension, M Crispen, December 2005
RFC 4467 Internet Message Access Protocol (IMAP) - URLAUTH extension. M. Crispin, May 2006
RFC 4469 Internet Message Access Protocol (IMAP) - CATENATE extension. P. Resnick, April 2006
RFC 4550 Internet Email to Support Diverse Service Environments (Lemonade) Profile. S. Meas, A. Melnikov, June 2006
RFC 4551 IMAP Extenstion for Conditional STORE Operation of Quick Flag Changes Resyncronization. A. Melnikov, S.Hole, June 2006
RFC 4731 IMAP4 extension to SEARCH command for controlling what kind of information is returned, A. Melnikov, D. Cridland, November 2006
RFC 4959 The IMAP Extension for SASL Initial Client Response. Robert Siemborski, A. Gulbrandsen Sept 2007
RFC 4978 The IMAP COMPRESS Extension. Arnt Gulbandsen, August 2007
RFC 5032 WITHIN Search extensions to the IMAP Protocol. Eric Burger, September 2007
RFC 5092 IMAP URL Scheme. A. Melnikov, C Newman, November 2007
RFC 5161 The IMAP ENABLE Extension. A. Melnikov, A. Gulbrandsen, March 2008
RFC 5162 IMAP4 Extensions for Quick Mailbox Resynchronization, A. Melnikov, D. Cridland, C. Wilson, March 2008
RFC 5228 Sieve: A Mail Filtering Language. T. Showalter, January 2008
RFC 5230 Sieve Email Filtering: Vacation Extension. T. Showalter, N. Freed, January 2008
RFC 5231 Sieve Email Filtering: Relational Extension. W. Segmuller, B. Leiba January 2008
RFC 5233 Sieve Email Filtering: Subaddress Extension. K. Murchison, January 2008
RFC 5256 Internet Message Access Protocol - SORT and THREAD Extensions, M. Crispin, K. Murchison, June 2008
RFC 5267 Contexts for IMAP4, D. Cridland, C. King, July 2008
RFC 5258 IMAP4 LIST Command extensions, B. Leiba, A. Melnikov, June 2008
RFC 5260 Sieve Email Filtering: Date and Index Extensions. N. Freed, July 2008
RFC 5804 A Protocol for Remotely Managing Sieve Scripts. T. Martin, A. Melnikov, July 2010
RFC 6152 SMTP Service Extension for 8-bit MIME transport, J. Klensin, N. Freed, M. Rose, E. Stefferud, D. Crocker, March 2011
RFC 6154 IMAP LIST Extension for Special-Use Mailboxes, B. Leiba, J. Nicolson, March 2011
RFC 6851 Internet Message Access Protocol (IMAP) - MOVE Extension, A. Gulbrandsen, N. Freed, Ed. Oracle, January 2013
Draft The IMAP SEARCH=INTHREAD and THREAD=REFS Extensions, Arnt Gulbrandsen, July 2010