Messaging and Directory Server software used around the world in the Government, Military, Aviation and Commercial sectors.

isode.com / whitepapers /

Scaling Isode's Messaging Solution

8th December 2006

Messaging service providers need to deal with very large numbers of messages and mailboxes. The underlying systems need to be able to handle these heavy loads and be able to grow as customer and traffic volumes increase. From the service provider's perspective, the messaging system should obey two 'golden rules':

1. It should not act as a barrier to future growth.
2. It should not require the purchasing of extra capacity in advance of that future growth.

This paper describes how Isode's messaging solution has been designed to meet these rules by scaling in a way that minimises deployment costs and maintenance.

 

Isode's Overall Architecture

Isode's service provider messaging solution is based on three server products and their associated management tools:

• M-Switch: SMTP (Simple Mail Transfer Protocol) Message Switch with Anti-Spam and Anti-Virus capabilities.
• M-Box: IMAP/POP message store with support for the latest mobile messaging capabilities. M-Box enables efficient, standards-based desktop and mobile email services.
• M-Vault: LDAP directory server supporting client authentication and authorization, as well as application and email configuration. Directory is a critical infrastructure component for service providers (see our whitepaper "Why do ISPs need LDAP?" for more information).

Horizontal Scaling & Load Balancing

Isode's approach to scaling is an approach generally known as "horizontal scaling". Multiple servers are used to provide the service, and if more capacity is needed, more servers are added. This approach makes scaling very simple and means that there is no requirement to buy special hardware for larger deployments, keeping the 'per server' cost reasonable.

The messaging service is provided by multiple servers and load balancers, although the structure of the configuration is invisible to the end user (as shown in the diagram below).

Load balancers are standard network components, and are the best way to distribute load between multiple servers. Lower cost approaches, typically based on DNS, do not generally lead to even load balancing.

Products with Natural Scaling

Three of the four components of the Isode messaging solution have a very natural horizontal scaling, so that capacity can be increased by adding more servers. In larger deployments, it is likely that there will be one Isode server per box, so that one server will run M-Vault or M-Switch or M-Box or Webmail. In a smaller system running multiple products on one server is quite feasible.

Message Switch (M-Switch)

The two diagrams below illustrate two variations on horizontal scaling of M-Switch, and show how additional M-Switch servers can be added to increase capacity. The inbound load balancer shares inbound SMTP traffic transparently between the M-Switch servers. Outbound SMTP traffic (typically arising from SMTP submission of messages from customers to non-local recipients) does not need to be load balanced.

Variation 1 shows M-Switch scaling managed independently of M-Box scaling. Here, LMTP delivery traffic from M-Switch to M-Box is shared using a load balancer.

In Variation 2 each M-Switch server has an assigned M-Box server, so there is no requirement for the second load balancer. Multiple M-Switch servers may be configured to deliver to the same M-Box server.

WebMail

Webmail based on IMAP and SMTP has natural horizontal scaling. Messages and "state" are held in the IMAP server, so there is no requirement for Webmail processes to share information in other ways. Webmail will run as an application in the front end Web servers (application servers).

Increased Webmail capacity can be achieved by adding more Webmail (Web) servers. Each Webmail server can be associated with specific IMAP and SMTP servers, or can share access to multiple servers using a load balancer, as shown above.

Directory (M-Vault) and Provisioning

The primary role of the M-Vault LDAP Directory is the provision of configuration, authentication and routing information to the other Isode servers. It can also be used for white pages services, including capabilities such as certificate lookup for end users.

In the configuration shown above, updates are provided to a single master directory, ensuring guaranteed updates for changes. These will be supplied by a provisioning system, which may be a complex database, or simply a Web front end onto the directory.

In a small system, the master directory may also provide LDAP service. In a larger system, as illustrated above, the M-Vault master will replicate data to multiple shadow directory servers. LDAP access to the shadow servers is load balanced across the shadow servers. Increased message traffic will lead to increased LDAP access. LDAP capacity can be increased by adding additional shadow servers.

Message Store (M-Box) Scaling

In this section we'll look at why scaling for Message Stores such as M-Box is inherently harder than for the other products. We’ll examine the "two tier" architecture promoted by some vendors (and its problems) as well as Isode's approach to Message Store scaling.

Why Message Store Scaling is Hard

With many servers, capacity is added by simply adding more servers. This does not work for a message store, where the mailboxes are associated with a specific server. When user X connects, and needs to access his/her mailbox, the connection needs to be to the specific server holding that mailbox, and not to an arbitrary server. This breaks the simple horizontal scaling model where service can be provided by any server.

This architectural problem is compounded by rapid growth in mailbox sizes. A typical service provider will want to offer 2 GBytes of message storage per user. This means that a service provider will need a lot of disk, and this impacts the scaling problem.

Why a Two-Tier Server Architecture is Bad

In a two tier architecture model, the mailbox service is provided by multiple back end servers, each supporting a number of mailboxes. In order to support more mailboxes, more back end servers are added. In order to connect users to the correct mailbox, a tier of front end servers are used. The user will connect to any (load balanced) front end server, where the IMAP or POP connection will be authenticated. The front end server will then connect to the appropriate back end server, and from then on "sit in the middle" of the connection. The front end server will also deal with situations where a client needs to access mailboxes on multiple servers.

This architecture is promoted by a number of vendors. It is a poor choice for a number of reasons:

1. Having an intermediate server that handles the connection is wasteful of resource, as you need two servers for each connection.
2. Allocating new mailboxes between servers can be tricky. It is important to share disks sensibly, and to ensure that loading is even between back end servers.
3. When disks become full, mailboxes need to be migrated to different back end servers. This process adds management and operational complexity.
4. If a back end server fails, the mailboxes that are supported by that server become unavailable.

Although it looks pretty, a two tier message store architecture has serious drawbacks.

Isode's Approach

Isode’s approach to horizontal scaling of M-Box is a single tier approach, that gives natural horizontal scaling. Mailboxes all reside on a single network file store, with uniform naming. This simple approach removes the problems of the two tier architecture.

The thinking behind this is that there are two elements to scaling that need to be dealt with: Message Store Scaling; and Disk Scaling.

There are many excellent commercial solutions to disk scaling (such as those from Network Appliance Inc and EMC), and Isode’s approach is to make use of these rather than to re-invent them. A network storage solution will provide the back end needed by a horizontally scaled M-Box solution, and will provide its own natural scaling to allow addition of disks. Filestore management, and in particular backup handling are a part of the network fileserver solution.

M-Box Features

The single tier architecture leads to a number of product requirements that are not immediately obvious. These requirements are the reason why the single tier architecture is not proposed by other vendors. Both requirements arise because messages may be delivered on different servers to where they are accessed, or may be accessed at the same time from more than one server.

1. Message delivery (or modification) and access may happen at the same time. M-Box has been designed with a very simple back end mailbox format that stores one message per file and makes direct use of the underlying file store. There are no complex indexes or database structures, meaning that no special locking is needed; M-Box relies on the underlying file system. Use of an index or database would generally require file or database locking, a significant performance hit. There is also risk of mailbox corruption, which the M-Box approach avoids.
2. For IMAP (or Webmail) access, the client needs to be notified when new messages arrive, whichever server it is delivered on. M-Box provides an efficient event mechanism, that notifies clients of new messages but does not waste resource when messages are delivered to a mailbox with no active client.

Conclusions

Isode's messaging solution is built using high performance servers, which can scale horizontally to support very large deployments.