Messaging and Directory Server software used around the world in the Government, Military, Aviation and Commercial sectors.
M-Switch
SMTP is a high performance, highly flexible and robust Internet Message
Switch using directory configuration.
Deployment Targets
M-Switch has two basic deployment targets described below: Boundary Message Switching and ISP.
Boundary Messaging
The boundary configuration shown above is where M-Switch SMTP performs an SMTP (Simple Mail Transfer Protocol) message switching. This boundary messaging configuration will often be used in conjunction with a firewall, to control inbound and outbound messages, and to provide anti-spam and anti virus capabilities described in more detail under M-Switch Anti-Spam.
This boundary switch may route internally to multiple departmental systems, and may do address rewriting to provide a uniform external appearance. M-Switch can use LDAP to access multiple departmental directories to perform boundary address validation.
ISP
M-Switch is the message switching component of Isode’s ISP solution. Details of the whole solution are given on the ISP Solutions page.
Key Benefits of M-Switch SMTP
The M-Switch has many functions, which are described below. Reasons why this product may be of particular interest include:
Flexibility
The architecture of the Message Switch, the management tools, and directory based configuration combine to give a very high degree of customer flexibility.
Performance
M-Switch SMTP has very high throughput and low latency.
Excellent scheduling and operational characteristics
The Queue Manager (QMGR) and channel architecture described below enables a sophisticated scheduling approach, which combined with the Message Switch's queue structure leads to a product which works exceedingly well in demanding operational environments. More details are given here.
Rock Solid
M-Switch has exceptional robustness and stability, including support for fail-over clustering and Off Site Hot Standby (Disaster Recovery).
Mobile Messaging Support
M-Switch SMTP supports the Lemonade profile.
LDAP directory based configuration
The Message Switch uses LDAP as its preferred configuration mechanism,
which enables sharing of routing and configuration information, and
flexible client/server management. Mailbox information is also held
in LDAP directories, which can be integrated with an ISP provisioning
system, or make use of existing (departmental) LDAP directories.
|
Management Features
The product has a wide range of management features, including configuration, SNMP monitoring, distribution lists, content conversion, message tracking, statistics, quarantine management and address mapping control. For more information see the Message Operator Interface page.
Address Handling
There are sophisticated features to deal with complex email addressing situations, including alias and redirect capabilities and address mapping mechanisms.
Security
There are a variety of security features, including SASL verified submission, TLS data confidentiality, audit database, and a general purpose message authorization control.
Managing M-Switch SMTP
M-Switch SMTP has a number of leading edge management features which include:
- Directory (LDAP) based configuration.
- Flexible address mapping and manipulation.
- Message authorization and access control.
- Audit database, statistics and tracking.
Managing M-Switch SMTP is described separately from this page in two places:
- The tools for monitoring and controlling an operational system are described on the Messaging Operational Management and include audit database, message tracking and message archiving.
- Configuration of M-Switch SMTP is described on the Messaging Configuration Management page.
- For an ISP configuration, user information is held in an LDAP directory and this information is shared by M-Box and M-Switch. User management is handled by the Internet Messaging Administrator.
The rest of this page looks at the core M-Switch SMTP functionality.
Message Switch Functionality
|
Architecture
The overall M-Switch SMTP architecture and the primary processes that comprise M-Switch SMTP are illustrated in the diagram to the right (click on the thumbnail for the full version), and the following sections summarize how these components work and interact. The event handling system is described later, but not illustrated in this diagram.
Message Queue
The (on disk) message queue is the centre of M-Switch. Arriving messages are written to the disk and secured, which is central to reliable store and forward messaging. Messages may then be checked and processed locally, before being sent on and removed from the disk. Typically, messages will remain in the queue for a very short while, often less than a second.
Channels
Most M-Switch processes are channels, shown in purple on the diagram above. The exceptions are QMGR (which is special) and three processes associated with the audit database. Channels perform many functions, including bringing messages into M-Switch (e.g., SMTP Server), delivering or transferring messages out of M-Switch, and processing messages within M-Switch.
Queue Manager (QMGR)
QMGR is the operational core of M-Switch, communicating with channels and with management interfaces. Key QMGR tasks:
- Starting and stopping channel processes.
- Scheduling active processing and delivery of messages.
- Controlling number of connections to the SMTP server.
- Receiving status information from channels, on connections and message transfer status.
- Responding to information and control requests from management processes.
To achieve this, QMGR makes use of three protocols:
- An internal channel communication protocol.
- AgentX to enable SNMP monitoring (not included in R12.0)
- SOM (Switch Operational Management Protocol), which is used by Isode’s management tools and provides strong authentication and data confidentiality using TLS and SASL. SOM Client Libraries are available for use by third party management tools (example here).
QMGR is a multi-threaded event driven process that holds an internal representation of the status of all queued messages, channels and connections. By handling all of this with a single process that does not need to look at the queues on disk, very sophisticated scheduling, control and status reporting is available. QMGR is designed to efficiently manage very large message queues (hundreds of thousands of messages). All information held by QMGR and control options are available through the MConsole tool, described here.
SMTP & LMTP
Isode provides full support for Simple Mail Transfer Protocol (SMTP) as specified in RFC 2821, conforming to the Internet host requirements for messaging (RFC 1123) including message submission (RFC 4409). SMTP support includes a number of auxiliary Internet Standard related to general capabilities and Lemonade support. A full list is given in the conformance section at the end of this document.
LMTP (RFC 2033) is a variant of SMTP optimized for delivery to local message stores, also supported by Isode’s M-Box IMAP/POP server.
Message Transfer by SMTP out of M-Switch is provided by the SMTP channel and Message Delivery by LMTP is provided by the LMTP channel. M-Switch SMTP may run many instances of each of these channels.
Inbound SMTP is handled by a single multi-threaded SMTP Server process (channel), so that new SMTP connections do not require new processes.
Message submission uses the SMTP protocol. The Internet Standard for Message Submission (RFC 4409) gives a special port for this (587), distinct from the SMTP port used for message transfer (25). Message submission is frequently authenticated, whereas this is unusual for message transfer. The SMTP Server can be configured to listen on both ports, or two SMTP servers can be used (one for each port).
Message Submission Library
The Message Submission Library provides a standard internal service for putting messages onto the message queue, which ensures that per message functions such as authentication, routing, and processing calculations are applied consistently and correctly. Key functions provided by the submission library are:
- Sender validation.
- Recipient validation and message routing.
- Determining what internal processing is needed.
- Policy and authorization checks.
- Writing the message to disk.
The Message Submission Library is used in many components of M-Switch, and in particular in the SMTP Server.
M-Switch Configuration
M-Switch SMTP generally uses directory configuration, which has many advantages. Table based configuration is also provided, as it is useful for some specialized deployments.
The M-Switch GUI for managing directory configuration is EMMA, which is described in the Messaging Configuration Management page.
M-Switch configuration is used by all of the M-Switch processes. Logically, configuration information from the directory is used by all processes. As changes to configuration are generally infrequent, this is implemented by QMGR obtaining configuration information from the directory using LDAP, checking for updates, and sharing it with the other processes.
Self Test
In order to optimize performance, the number of channels running, number of each channel type running, number of connections to a single MTA, will vary dependent on many factors, including remote system, network and local hardware performance. Manual optimization is complex, and in any event the optimal configuration is likely to vary over time. For this reason, M-Switch does a sophisticated “self test”, to measure performance and adapts configuration accordingly. This enables M-Switch to adapt its operational parameters to different hardware and network environments.
Message Routing
Routing of messages to other SMTP servers uses the domain part of the email address and is generally handled using the DNS (Domain Name System). M-Switch also allows for connections to be configured using EMMA to specific SMTP hosts. Reasons for doing this include:
- Internal routing that over-rides DNS based routing.
- Configuring security options, such as TLS for data confidentiality and SASL for peer authentication using passwords or strong authentication.
- Configuring one or more permanent SMTP connections, which will reduce message switching latency by removing the time needed to establish a connection.
M-Switch also routes based on the whole email address. This is referred to as LASER routing, after the Internet working group that developed a specification with this name that did not get published as a standard. M-Switch broadly follows that specification, with a number of extensions, and options to deal with directory servers that store information in a different way.
The key concept in LASER routing is that you search in an LDAP directory for a mailbox of the form joe.user@mycorp.com. Routing succeeds when an entry is correctly matched. Additional attributes in the entry can control with channel and host the message is then sent to. Typically, the match will use a default LMTP connection to a local M-Box server.
Three basic ways that LASER routing may be used in conjunction with domain based routing:
- Single domain system, such as a consumer ISP or enterprise. If the domain is local, LASER routing is used, otherwise domain routing is used.
- Hosted service, handling multiple domains on one server. LASER routing is used to a single directory server. If there is a match, the message is handled according to the match. If not, domain routing is used for authorized message submission.
- Boundary switching to multiple departmental servers. Domain routing looks to identify locally configured domains. A match will identify the departmental LDAP server against which to do the LASER lookup, and the SMTP server to which matched addresses should be routed. This approach allows the boundary switch to fully validate email addresses by making use of existing LDAP servers.
M-Switch has flexible configuration that can support these setups and others. In addition to the core LASER routing, M-Switch also provides:
- A mechanism to specify alternate mailbox addresses.
- A list channel, that expands managed distribution lists.
- Sub-addressing support (joe+XXX@myorg.com).
Event Handling
M-Switch uses the Isode event handling system. Multiple event streams can be configured, with event access using Isode’s Event View, Unix Syslog, or Windows Event Manager. Isode events are documented with description and severity.
More details on the event system are given in this whitepaper on operational monitoring.
Audit Logging & Audit Database
Detailed information on messages handled and processing status are recorded in an audit log. This audit log is processed by the Message Audit Service which is stored in an audit database. The audit database may be local or remote and used by one or more M-Switch servers.
The Audit Database is accessed by MConsole and the Message Operator Interface which use this information to provide:
- Statistics and reports on M-Switch usage and performance.
- Message Tracking.
- Message Archive Retrieval.
- Quarantine management.
Further details are given in Message Operator Interface Overview.
Anti-Spam and Anti-Virus Channels
Detailed information on anti-spam and anti-virus are given under M-Switch Anti-Spam and M-Switch Anti-Virus. This section summarizes how these capabilities are provided in M-Switch. A number of anti-spam measures are implemented in the SMTP Server. Although messages may be rejected at the SMTP level, the SMTP server usually simply records the results of anti-spam checks in the message, so that they can be used in conjunction with other checks done in the anti-spam channel.
The core anti-spam function is performed by the Anti-Spam channel, using Isode developed technology. Isode spam checking does not use Spamassassin or other third party anti-spam technology. The anti-spam channel records its actions in audit logs, and may delete messages or move them from the message queue to the quarantine.
Users and operators may find information about messages in the quarantine
using the MOI, or by email quarantine summaries sent out by the Quarantine
Email Generator process, based on information in the audit database.
A user or operator can use MOI to mark a message in the audit database
to be released from quarantine. The Quarantine Release Process monitors
for messages to be released, and does this by re-submitting the quarantined
message.
The Anti-Virus channel operates in a similar manner, except that checks
are performed using a third party AV checker. It is straightforward
to use most AV checkers in conjunction with M-Switch.
File Transfer by Email
M-Switch provides infrastructure to transfer files by email, which is designed to support applications such as directory and database replication. It could also be used to support user file transfer.
There is a file transfer by email server that watches configured directories, and transfers files to one or more email recipients associated with each directory originating from an email address associated with the directory. Files are used with naming conventions to indicate message submission and successful or failed delivery to each recipient (using X.400 delivery reports). There is a matching file transfer by email channel that delivers messages and reports.
More information is given in the Isode white paper File Transfer by Email, and information on its use for Directory Replication is given in the white paper Directory Replication by Email and over 'Air Gap'.
Message Archive
Messages may be archived at the same time as they are placed into the M-Switch message queue. Messages are archived in a single file per message, in a file system directory. The archive directory is named using date/time, and changes at configurable intervals. This enables the archive directory to remain a reasonable size and to facilitate offline backup. The archive file name is audit logged along with other standard message audit information.
Message Tracking from Message Operator Interface Web Application or MConsole can use information in the audit database to locate messages, and then use the SOM protocol to retrieve archived messages via the QMGR (which directly accesses the identified file in the archive), and display or forward them. Similarly, these tools can also view messages in the queue and quarantine.
Message Delivery Options
As well as standard delivery to M-Box using LMTP, M-Switch supports a number of other message delivery options. Messages may be delivered using a Local Channel to local mail files, either in standard UNIX formats or using the popular maildir format. Delivery time processing includes a sophisticated delivery processing mechanism, controlled by a special language which defines the processing based on a wide range of message parameters. This can be used for personal functions such as automatically filing messages on delivery or sending holiday notifications. It is also useful to support system delivery functions.
The Program Channel (or Shell Channel) enables delivery of messages to arbitrary programs. This can be used to easily integrate message delivery with special purpose delivery functions. This gives a general purpose mechanism for building system specific delivery functions and automatic mailboxes (e.g., “echo” or “delete”).
Housekeeper Channel
A number of general functions are handled by a special Housekeeper Channel, including:
- Processing Delayed Messages. During message submission, DNS and LDAP are used to determine routing and processing of messages. In the event of a timeout on one of these services, the message is “delayed” and message submission is completed later by the Housekeeper Channel.
- Message Warning and Timeout. When a message has been in the queue for a configurable period, it will be timed out, and a delivery status notification sent to the sender. At intermediate times, messages warning of delay may also be sent.
- Operator requested Message Deletion and Non-Delivery. In normal processing, message deletion is handled by the last channel doing work on the message. The operator may request message deletion or non-delivery using MConsole, which will be done by the Housekeeper Channel.
- Message Redirect. The operator may request redirection of a message recipient to a new recipient using MConsole, which is done by Housekeeper Channel.
- Message Reprocessing. Routing configuration for M-Switch may change in a way that would cause different routing for a message. The operator may request that a message or groups of messages are “reprocessed”, so that the updated routing calculation is applied. This is particularly important if a routing change is made to enable queue messages to take an alternate route.
Header and Message Content Processing
Internet messages are structured according to the MIME (Multipurpose Internet Message Extensions) Standard defined in RFC 2045. M-Switch has a MIMEShaper channel that understands MIME format and can perform conversion on messages and components within messages.
Individual body parts can be converted from one to another using conversion filters. Typically this is used for converting a text body part from one character set to another. The necessary conversions are calculated when a message is first submitted and they may be re-evaluated when a message is 'exploded'.
It is often desirable to rewrite header information in particular to 'normalize' addresses by rewriting the address in some canonical form, rather than one of the multiple addresses that can be used to reach a specific recipient. MIMEShaper provides options for the header normalization of Internet message headers are provided. This capability can be used to provide a coherent view of addresses for local users, or to manage addresses to give an external view in a firewall type configuration.
External Content Processing (CCCP)
M-Switch has the capability to use customer or partner provided capabilities to check and convert Internet messages. Although M-Switch provides a comprehensive range of checking and conversion capabilities, there will sometimes be customer requirements that go beyond this. This is achieved by use of a protocol called CCCP (Content Checking and Conversion Protocol). The CCCP specification is available here.
M-Switch uses CCCP to provide information to a customer developed CCCP server, that performs checking and conversion of messages. CCCP is a simple text encoded protocol with an encoding approach aligned with the IMAP protocol. CCCP enables the following functionality to be provided:
- Recipient changing (can be regarded as re-writing or redirect).
- Sender re-write.
- Recipient expansion (so the CCCP server can implement simple lists).
- Content conversion. The CCCP server may provide back modified messages, and different copies can be assigned to different recipients.
- For each recipient, message can be non-delivered, discarded, or quarantined (the CCCP requests M-Switch to perform the action).
These features allow a CCCP server to perform a wide range of address and content checking and conversion functions. The client/server architecture gives a number of advantages:
- CCCP processing is applied to stored messages, so this is completely transparent to external message transfer.
- Parallel processing means that slow CCCP processing of one message (e.g., taking human intervention) will not delay other message processing.
- Conversion and Checking is cleanly decoupled from M-Switch.
- For large deployments, M-Switch and CCCP servers can be independently horizontally scaled.
M-Switch provides flexible use of CCCP, so that multiple CCCP channels can be configured to provide different conversions for different message flows. CCCP enables a wide range of conversion capabilities for different markets, and Isode is happy to work with any partner or customer to develop new CCCP servers.
Message Priority
M-Switch handles messages based on priority, which is available for all protocols (not just X.400). Message priorities for Internet messages are determined from MIXER "Priority:" Headers. The List Channel may downgrade messages to a "bulk" priority, and Anti Spam channel may downgrade messages to "junk" priority. M-Switch has a number of capabilities for handling messages of different priorities, described in more detail on the M-Switch X.400 product page.
Authorization & Routing Policy
M-Switch provides a comprehensive authorization framework, for controlling which messages can be sent. This includes controls for individual senders and recipients. Controls are often applied on the basis of channel. So far channels have been described in functional terms. Multiple logical channels can be created using the same protocol. For example to create “SMTP Internal” and “SMTP External” channels, each with a group of associated MTAs. This logical grouping can be used to set up authorization policy, which will enable appropriate message relaying and block other types of access.
Security and Authentication
Isode uses Transport Layer Security (TLS) for data confidentiality and Simple Authentication and Security Layer (SASL) for authentication. SASL is also used to map simple identifiers onto directory names for authentication. X.509 (PKI) based authentication can be used, by using TLS and SASL-EXTERNAL authentication. SASL and TLS are supported in the SMTP and LMTP protocols. This is particularly important for controlling message submission, and may also be configured for server to server communication.
SASL and TLS are used by Isode’s SOM protocols, so that secure management from MConsole is possible.
S/MIME
M-Switch can parse S/MIME (RFC 3851) messages so that information encoded as S/MIME is available for conversion and checking capabilities. In particular, this enables anti-virus and anti-spam content checks on (non-encrypted) S/MIME encoded messages.
M-Switch can encode outgoing messages as S/MIME, which will be signed by M-Switch. M-Switch can also add a signature to an S/MIME message that is being relayed.
Security Labels
Modern use of Security Labels will involve a comprehensive structured format such as the ESS format defined in RFC 2634 "Enhanced Security Services for S/MIME". Other techniques are also used for supporting Security Labels in email, such as FLOT (First Line of Text) labels. This is an example FLOT label:
Protective Marking: UNCLASSIFIED
M-Switch supports ESS labels, which it extracts from the S/MIME encoding.
The following ad hoc label formats are supported:
- FLOT (First Line of Text) for Internet Email and X.400, as the first line of text in the first text body part. This is supported for all types of text body part.
- Security label included within the Subject: line of an Internet email.
- Security Label in a special X- Header of an Internet email. This can be configured for any header.
A primary goal of Isode's mapping of Security Labels in messages is to convert between structured and ad hoc forms. Ad hoc labels are often surrounded by special (delimiting) text, and M-Switch can parse and generate this. Ad hoc labels are generated from structured labels by use of capabilities from Isode's Security Policy infrastructure.
Text labels can be generated from the Security Classification or from the entire Security Label, using a choice of marking data from the Security Policy used. There is no standardized approach for the reverse mapping. Isode provides a flexible text lookup and conversion mechanism to map from the (ad hoc) text security labels to Isode XML format structured labels. This approach gives a high degree of flexibility to support different ad hoc security label formats.
Latency
M-Switch has been designed to give low transfer latency as well as high message throughput.
Internet Mail Conformance
| RFC 1123 | Requirements for Internet hosts - application and support, R. Braden, October 1989 |
|---|
| RFC 1652 | SMTP Service Extension for 8-bit MIME transport, J. Klensin, N. Freed, M. Rose, E. Stefferud, D. Crocker, July 1994 |
|---|
| RFC 1870 | SMTP Service Extension for Message size Declaration, J. Klensin, N. Freed, K. Moore, November 1995 |
|---|
| RFC 2920 | SMTP Service Extension for Command Pipelining, N. Freed, September 2000 |
|---|
| RFC 2033 | Local Mail Transfer Protocol, J. Meyers, October 1996 |
|---|
| RFC 2034 | SMTP Service Extension for Returning Enhanced Error Codes, N Freed, October1996 |
|---|
| RFC 2045 | Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies, N. Freed, N. Borenstein, November 1996 |
|---|
| RFC 2046 | Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types, N. Freed, N. Borenstein, November 1996 |
|---|
| RFC 2047 | (Multipurpose Internet Mail Extensions) MIME Part Three: Header Extensions for Non-ASCII Text, K. Moore, November 1996 |
|---|
| RFC 2049 | (Multipurpose Internet Mail Extensions) MIME Part Five: Conformance Criteria and Examples, N. Freed, N. Borenstein, November 1996 |
|---|
| RFC 2476 | Message Submission, R. Gellens , J. Klensin December 1998 |
|---|
| RFC 2505 | Anti-Spam Recommendations for SMTP MTAs, G. Lindberg, February 1999 |
|---|
| RFC 2634 | Enhanced Security Services for S/MIME, P. Hoffman, June 1999 |
|---|
| RFC 2821 | Simple Mail Transfer Protocol, J. Klensin (ed), April 2001 |
|---|
| SMTP Service Extensions for Transmission of Large and Binary MIME Messages, G. Vaudreuil, December 2000 |
| SMTP Service Extension for Secure SMTP over Transport Layer Security, P. Hoffman, Feb 2002 |
| Cryptographic Message Syntax (CMS), R. Housley, Auguest 2002 |
| RFC 3461 | Simple Mail Transfer Protocol (SMTP) Service Extension for Delivery Status Notifications (DSNs), K. Moore, January 2003 |
|---|
| RFC 3462 | The Multipart/Report Content Type for the Reporting of Mail System Administrative Messages, G. Vaudreuil, January 2003 |
|---|
| RFC 3848 | ESMTP and LMTP Transmission Types Registration, C. Newman, July 2004 |
|---|
| Secure/Multipurpose Internet Mail Extensions (S/MIME) Version 3.1 Message Specification, B. Ramsdell, July 2004 |
| RFC 4468 | Message Submission BURL Extension, C. Newman, May 2006 |
|---|
| RFC 4954 | SMTP Service Extension for Authentication, A. Melnikov, R. Siemborski July 2007 |
|---|
Availability
M-Switch is available on Solaris, Windows, Linux and HP-UX. More details on supported platforms and versions can be found here.
