M-Switch architecture: click to reveal details
The (on disk) message queue is the centre of M-Switch. Arriving messages are written to the disk and secured, using fsync() to ensure no possibility of message loss, which is central to reliable store and forward messaging. Messages may then be checked and processed locally, before being sent on and removed from the disk. Typically, messages will remain in the queue for a very short while, often for just a few milliseconds.
Most M-Switch processes are channels, shown as purple rectangles on the diagram above. The exceptions are QMGR (see below) and three processes associated with the audit database. Channels perform many functions, including bringing messages into M-Switch (e.g. SMTP Server), delivering or transferring messages out of M-Switch, and processing messages within M-Switch.
Queue Manager (QMGR)
QMGR is the operational core of M-Switch, communicating with channels and with management interfaces. Key QMGR tasks:
- Starting and stopping channel processes.
- Scheduling active processing and delivery of messages.
- Controlling number of connections to the SMTP server.
- Controlling the number of connections to peer MTAs.
- Receiving status information from channels, on connections and message transfer status.
- Responding to information and control requests from management processes.
- Ensuring that higher priority messsages (3 civilian priorities or 6 military priorities) are handled first.
- Reads the MTA configuration in the Directory and downloads it into the mtatailor file for other processes (such as channels) to read. Some configuration (such as routing), is read directly from the DSA by M-Switch processes.
To achieve this, QMGR makes use of three protocols:
- An internal channel communication protocol.
- AgentX to enable SNMP monitoring.
- SOM (Switch Operational Management Protocol), which is used by Isode’s management tools and provides strong authentication and data confidentiality using TLS and SASL. SOM Client Libraries are available for use by third party management tools (examples in the [Isode Servers and Sentra] whitepaper).
QMGR is a multi-threaded event driven process that holds an internal representation of the status of all queued messages, channels and connections. By handling all of this with a single process that does not need to look at the queues on disk, very sophisticated scheduling, control and status reporting is available. QMGR is designed to efficiently manage very large message queues (hundreds of thousands of messages). All information held by QMGR and control options are available through the MConsole tool.
SMTP & LMTP
Isode provides full support for Simple Mail Transfer Protocol (SMTP) as specified in RFC 5321, conforming to the Internet host requirements for messaging (RFC 1123) including message submission (RFC 6409). SMTP support includes a number of auxiliary Internet Standard related to general capabilities and Lemonade support. A full list is given on the M-Switch conformance page.
Message Transfer by SMTP out of M-Switch is provided by the SMTP channel and Message Delivery by LMTP is provided by the LMTP channel. M-Switch may run many instances of each of these channels.
Inbound SMTP is handled by a single multi-threaded SMTP Server process (channel), so that new SMTP connections do not require new processes.
Message submission uses the SMTP protocol. The Internet Standard for Message Submission (RFC 6409) gives a special port for this (587), distinct from the SMTP port used for message transfer (25). Message submission is frequently authenticated, whereas this is unusual for message transfer. The SMTP Server can be configured to listen on both ports, or two SMTP servers can be used (one for each port).
Security for SMTP includes TLS (using START-TLS) and SASL based authentication, supporting a wide range of security options including client strong authentication using SASL EXTERNAL.
Message Submission Library
The Message Submission Library provides a standard internal service for putting messages onto the message queue, which ensures that per message functions such as authentication, routing, and processing calculations are applied consistently and correctly. Key functions provided by the submission library are:
- Sender validation.
- Recipient validation and message routing.
- Determining what internal processing is needed.
- Policy and authorization checks.
- Writing the message to disk.
The Message Submission Library is used in many components of M-Switch, and in particular in the SMTP Server.
M-Switch generally uses directory configuration. Table based configuration is also provided, as it is useful for some specialized deployments, particularly where a very simple static configuration is needed.
The M-Switch directory configuration is managed using the Switch configuration View in MConsole, which is described in the Messaging Configuration Management page. Table based configurations are managed by editing text files.
M-Switch configuration is used by all of the M-Switch processes. Logically, configuration information from the directory is used by all processes. As changes to configuration are generally infrequent, this is implemented by QMGR obtaining configuration information from the directory using LDAP, checking for updates, and sharing it with the other processes.
In order to optimize performance, the number of channels running, number of each channel type running, number of connections to a single MTA, will vary dependent on many factors, including remote system, network and local hardware performance. Manual optimization is complex, and in any event the optimal configuration is likely to vary over time. For this reason, M-Switch performs a sophisticated "self test", to measure performance and adapts configuration accordingly. This enables M-Switch to adapt its operational parameters to different hardware and network environments.
Routing of messages to other SMTP servers uses the domain part of the email address and is generally handled using the DNS (Domain Name System), but may also use locally configured host resolution. M-Switch also allows for connections to be configured to specific SMTP hosts using MConsole. Reasons for doing this include:
- Internal routing that overrides DNS based routing.
- Configuring security options, such as TLS for data confidentiality and SASL for peer authentication using passwords or strong authentication.
- Configuring one or more permanent SMTP connections, which will reduce message switching latency by removing the time needed to establish a connection.
M-Switch can also be set up to route based on the whole email address. This is referred to as LASER routing, after the Internet working group that developed a specification with this name (that did not get published as a standard). M-Switch broadly follows that specification, with a number of extensions, and options to deal with directory servers that store information in a different way.
The key concept in LASER routing is that you search in an LDAP directory for a mailbox of the form email@example.com. Routing succeeds when the search for the address returns in exactly one result for an Entry which holds suitable delivery information. Additional attributes in the entry can control with channel and host the message is then sent to. Typically, the match will use a default LMTP connection to a local M-Box server. This architecture works well for organizations with large “white pages” directories.
There are three basic ways that LASER routing may be used in conjunction with domain based routing:
- Single domain system, such as a consumer ISP or enterprise. If the domain is local, LASER routing is used, otherwise domain routing is used.
- Hosted service, handling multiple domains on one server. LASER routing is used to a single directory server. If there is a match, the message is handled according to the match. If not, domain routing is used for authorized message submission.
- Boundary switching to multiple departmental servers. Domain routing looks to identify locally configured domains. A match will identify the departmental LDAP server against which to do the LASER lookup, and the SMTP server to which matched addresses should be routed. This approach allows the boundary switch to fully validate email addresses by making use of existing LDAP servers.
M-Switch has flexible configuration that can support these setups and others. In addition to the core LASER routing, M-Switch also provides:
- An aliasing mechanism.
- A mechanism to specify alternate mailbox addresses.
- A list channel, that expands managed distribution lists.
- Sub-addressing support (joe+XXX@myorg.com).
M-Switch uses the Isode event handling system. Multiple event streams can be configured to report events to log files; Unix Syslog; Windows Event Manager; an XMPP chat room; or any combination. Events reported by an MTA can be accessed from the QMGR using the SOM protocol using MConsole's Event View. Isode events are documented with description and severity.
More details on the event system are given in this whitepaper on operational monitoring.
Audit Logging & Audit Database
Detailed information on messages handled and processing status are recorded in an audit log. This includes full details of messages, Delivery Status Notifications, and Message Disposition Notifications. This audit log is processed by the Message Audit Service which is stored in an audit database. The audit database may be local or remote and used by one or more M-Switch servers.
The Audit Database is accessed by MConsole, which use this information to provide:
- Statistics and reports on M-Switch usage and performance.
- Message Tracking.
- Message Archive Retrieval.
- Quarantine management.
The audit database also enables Message Correlation, to provide information derived from delivery reports and read receipts is provided by the MConsole Acknowledgement View and by the Quality of Service Daemon. For more information see the whitepaper [Using Message Acknowledgements for Tracking, Correlation and Fire & Forget].
Messages may be archived at the same time as they are placed into the M-Switch message queue. Messages are archived as a single file per message, in a file system directory. The archive directory is named using date/time, and changes at configurable intervals. This enables the archive directory to remain a reasonable size and to facilitate offline backup. The archive file name is audit logged along with other standard message audit information.
Messages may also be archived in a format that reflects the message sent. This is useful when M-Switch performs message format conversion, so that both formats can be archived and used in message tracking.
Message Tracking and Message History views in MConsole use information in the audit database to locate messages, and then use the SOM protocol to retrieve archived messages via the QMGR (which directly accesses the identified file in the archive), and display or forward them. This gives operator access to both inbound and outbound message archives.
M-Switch has the capability to “hold” messages. This can be applied to all messages, or to messages selected by an authorization rule (e.g., messages to a specific destination, or messages that fail a check such as digital signature validation). These can be released by protocol (SOM) or by an operators in a view in MConsole for releasing held messages.
M-Switch supports two SMTP protocols to control delivery timing:
- SMTP Control to deliver messages by a certain time following RFC 2852 "Deliver By SMTP Service Extension".
- SMTP Control to hold messages for delivery until a specific time following RFC 4865 "SMTP Submission Service Extension for Future Message Release".
Message Delivery Options
As well as standard delivery to M-Box using LMTP, M-Switch supports a number of other message delivery options. Messages may be delivered using a Local Channel to local mail files, either in standard UNIX formats or using the popular maildir format. Delivery time processing includes a delivery processing mechanism, controlled by a special language which defines the processing based on a wide range of message parameters. This can be used for personal functions such as automatically filing messages on delivery or sending holiday notifications. It is also useful to support system delivery functions. Modern systems are more likely to use LMTP delivery to a message store. Isode M-Box supports sophisticated delivery processing using SIEVE as specified in RFC 5228.
The Program Channel (or Shell Channel) enables delivery of messages to arbitrary programs. This can be used to easily integrate message delivery with special purpose delivery functions. This gives a general purpose mechanism for building system specific delivery functions and automatic mailboxes (e.g., 'echo' or 'delete').
A number of general functions are handled by a special Housekeeper Channel, including:
- Processing Delayed Messages. During message submission, DNS and LDAP are used to determine routing and processing of messages. In the event of a timeout on one of these services, the message is "delayed" and message submission is completed later by the Housekeeper Channel.
- Message Warning and Timeout. When a message has been in the queue for a configurable period, it will be timed out, and a delivery status notification sent to the sender. At intermediate times, messages warning of delay may also be sent.
- Operator requested Message Deletion and Non-Delivery. In normal processing, message deletion is handled by the last channel doing work on the message. The operator may request message deletion or non-delivery using MConsole, which will be done by the Housekeeper Channel.
- Message Redirect. The operator may request redirection of a message recipient to a new recipient using MConsole, which is done by Housekeeper Channel.
- Message Reprocessing. Routing configuration for M-Switch may change in a way that would cause different routing for a message. The operator may request that a message or groups of messages are "reprocessed", so that the updated routing calculation is applied. This is particularly important if a routing change is made to enable queue messages to take an alternate route.
The Housekeeper channel is started by the QMGR when M-Switch starts up to report information about messages in the queue back to the QMGR.
Message Priority & Latency
M-Switch has been designed to give low transfer latency as well as high message throughput.
M-Switch handles messages based on priority, which is supported for SMTP message transfer using RFC 6710 "Simple Mail Transfer Protocol Extension for Message Transfer Priorities". Message priorities for Internet messages are may also be determined from MIXER "Priority:" Header (three level X.400 priority) and from "MMHS-Primary-Precedence:" (six level military priority) specified in RFC 6477. The Anti Spam channel may downgrade messages to "junk" priority. Messaging management GUIs can display priority using military priority names. Message priority is set by the originating application, and may be modified by distribution list processing. M-Switch scheduling will always send higher priority messages first. Some or all channels may be set, using MConsole, to only send messages above a certain priority for a specified period. This can be used in support of military MINIMIZE.
M-Switch can also be configured to map between various Priority fields and Importance, for example to enforce consistency between various priority related fields.
Boundary Acknowledgement Support
M-Switch can be configured to generate acknowledgements for messages being relayed both read receipts (Message Disposition Notifications (MDNs)) and Delivery Status Notifications (DSNs). This is to support end systems which do not generate acknowledgements and security boundaries (e.g., data diode) where acknowledgements cannot be returned. There is an option to configure acknowledgement generation where they are not requested, and to control relayed acknowledgement requests.
M-Switch is an exceedingly fast SMTP switch. We published benchmarks for M-Switch handling of X.400 traffic in 2007. This showed sustained message switching at over 500 messages per second. SMTP switch is faster, due to lower protocol overheads, and we would also expect better results on new hardware.
The exact performance achieved will depend on a number of factors, including:
- Hardware choice
- Level of audit and event logging
- Use of Anti-Virus
- Message processing and formatting performed
We would recommend benchmarking with a desired configuration if throughput of greater than 100 messages per second is needed.
For highest performance we recommend use of an SSD drive for message queues.
Configuration Backup and Restore
Isode provides a backup and restore capability. This can be used to save a configuration and then to automatically generate variants of this configuration onto a clean install. This capability is to facilitate managed deployments of M-Switch and associated server products.