Follow Join Read

M-Switch SMTP

M-Switch SMTP is a high performance, highly flexible and robust Internet Message Switch using directory configuration. Its primary protocol support is SMTP (Internet Standard Simple Mail Transfer Protocol).

Deployment Configurations

M-Switch has two basic deployment configurations described below: Boundary and Full Messaging.

Boundary Messaging

 

Boundary Messaging

 

The boundary configuration shown above is where M-Switch SMTP performs an SMTP (Simple Mail Transfer Protocol) message switching. This boundary messaging configuration will often be used in conjunction with a firewall, to control inbound and outbound messages, and to provide anti-spam and anti virus capabilities described in more detail under M-Switch Anti-Spam.

This boundary switch may route internally to multiple departmental systems, and may do address rewriting to provide a uniform external appearance. M-Switch can use LDAP to access multiple departmental directories to perform boundary address validation.

Boundary configurations are often used to provide security services, including:

  • Security Label based Access Control and checking.
  • S/MIME digital signature singing and verification
  • Message archive, audit and tracking
  • Message content checking and conversion
  • Message authorization and rule based routing and checking
  • Interface to constrained networks, such as HF Radio and Satcom
  • File Transfer By Email over networks and gateways that only allow email.

For boundary messaging and support of mobile devices, M-Switch SMTP can be used in conjunction with M-Box Gateway.

M-Switch has advanced management and monitoring capabilities, which make it particularly suitable for high assurance managed boundary operation. For more information see the Internet Messaging Management page.

Full Messaging

 

Full Messaging Solution

 

M-Switch is the message switching component of Isode's Internet Messaging solution. This is used in conjunction with M-Box which provides message storage, and access by POP and IMAP. This can be used for various types of deployment.

  • As an ISP (Internet Service Provider) solution for fixed and mobile messaging. Details of the service provider solution are given on the ISP Solutions page.
  • As a small system, typically single user or with a small number of users, in a mobile deployment with deployment over constrained networks.
  • In a secure environment, where M-Switch's security and management features, such as Security Label Support, Archiving, and Message Tracking are needed in conjunction with messaging client access using Open Standards.

Military Messaging

A number of M-Switch Capabilities make it particularly suitable for Military Messaging.

Key Benefits of M-Switch SMTP

The M-Switch has many functions, which are described below. Reasons why this product may be of particular interest include:

Flexibility

The architecture of the Message Switch, the management tools, and directory based configuration combine to give a very high degree of customer flexibility.

Performance

M-Switch SMTP has very high throughput and low latency.

Excellent scheduling and operational characteristics

The Queue Manager (QMGR) and channel architecture described below enables a sophisticated scheduling approach, which combined with the Message Switch's queue structure leads to a product which works exceedingly well in demanding operational environments. More details are given here.

Rock Solid

M-Switch has exceptional robustness and stability, including support for fail-over clustering and Off Site Hot Standby (Disaster Recovery).

Mobile Messaging Support

M-Switch SMTP supports the Lemonade profile.

LDAP directory based configuration

The Message Switch uses LDAP as its preferred configuration mechanism, which enables sharing of routing and configuration information, and flexible client/server management. Mailbox information is also held in LDAP directories, which can be integrated with an ISP provisioning system, or make use of existing (departmental) LDAP directories.

Management Features

The product has a wide range of management features, including configuration, SNMP monitoring, distribution lists, content conversion, message tracking, statistics, quarantine management and address mapping control. For more information see the Message Operator Interface and Internet Messaging Management pages.

Address Handling

There are sophisticated features to deal with complex email addressing situations, including alias and redirect capabilities and address mapping mechanisms.

Security

There are a variety of security features, including SASL verified submission, TLS data confidentiality, audit database, and a general purpose message authorization control.

Security Labels and S/MIME

M-Switch supports S/MIME digitally signed messages, and can verify inbound messages and sign outgoing messages. It can also handle Security Labels which are carried use S/MIME Extended Security Services (ESS). M-Switch gives Access Control checks associated with Security Clearance of channels, peer MTAs and users. It can also convert security labels with other formats, including support of FLOT (First Line of Text) labels.

S/MIME Encryption is supported by M-Switch Encryption, which is a capability that may be added to M-Switch

Constrained Network Support, including HF Radio and Satcom

M-Switch provides optimized support for constrained networks, including HF Radio, faster Radio including VHF/UHF and Satcom. This includes operation over point to point and multicast networks and support of EMCON (Emission Control) where nodes are in radio silence.

Conversion

M-Switch provides a range of built in message format conversion capabilities, including S/MIME and Security Label handling. It also enables customer provided message checking and conversion using the CCCP (Content Conversion and Checking Protocol).

File Transfer by Email

M-Switch provides easy support for transfer of files over email to one or more destinations. This works by applications simply copying files to and from directories. It can also be used in conjunction with S/MIME and Security Labels.

Managing M-Switch SMTP

M-Switch SMTP has a number of leading edge management features which include:

  • MConsole GUI for configuration and operational management.
  • Directory (LDAP) based configuration.
  • Flexible address mapping and manipulation.
  • Message authorization and access control.
  • Audit database, statistics and tracking.

Managing M-Switch SMTP is described separately on the Internet Messaging Management page.

The rest of this page looks at the core M-Switch SMTP functionality.

Architecture

The overall M-Switch SMTP architecture and the primary processes that comprise M-Switch SMTP are illustrated in the diagram to the right (click on the thumbnail for the full version), and the following sections summarize how these components work and interact. The event handling system is described later, but not illustrated.

Message Queue

The (on disk) message queue is the centre of M-Switch. Arriving messages are written to the disk and secured, which is central to reliable store and forward messaging. Messages may then be checked and processed locally, before being sent on and removed from the disk. Typically, messages will remain in the queue for a very short while, often less than a second.

Channels

Most M-Switch processes are channels, shown in purple on the diagram above. The exceptions are QMGR (which is special) and three processes associated with the audit database. Channels perform many functions, including bringing messages into M-Switch (e.g. SMTP Server), delivering or transferring messages out of M-Switch, and processing messages within M-Switch.

Queue Manager (QMGR)

QMGR is the operational core of M-Switch, communicating with channels and with management interfaces. Key QMGR tasks:

  • Starting and stopping channel processes.
  • Scheduling active processing and delivery of messages.
  • Controlling number of connections to the SMTP server.
  • Controlling the number of connections to peer MTAs.
  • Receiving status information from channels, on connections and message transfer status.
  • Responding to information and control requests from management processes.

To achieve this, QMGR makes use of three protocols:

  • An internal channel communication protocol.
  • AgentX to enable SNMP monitoring.
  • SOM (Switch Operational Management Protocol), which is used by Isode’s management tools and provides strong authentication and data confidentiality using TLS and SASL. SOM Client Libraries are available for use by third party management tools (example here).

QMGR is a multi-threaded event driven process that holds an internal representation of the status of all queued messages, channels and connections. By handling all of this with a single process that does not need to look at the queues on disk, very sophisticated scheduling, control and status reporting is available. QMGR is designed to efficiently manage very large message queues (hundreds of thousands of messages). All information held by QMGR and control options are available through the MConsole tool, described here.

SMTP & LMTP

Isode provides full support for Simple Mail Transfer Protocol (SMTP) as specified in RFC 5321, conforming to the Internet host requirements for messaging (RFC 1123) including message submission (RFC 6409). SMTP support includes a number of auxiliary Internet Standard related to general capabilities and Lemonade support. A full list is given in the conformance section at the end of this document.

LMTP (RFC 2033) is a variant of SMTP optimized for delivery to local message stores, also supported by Isode’s M-Box IMAP/POP server.

Message Transfer by SMTP out of M-Switch is provided by the SMTP channel and Message Delivery by LMTP is provided by the LMTP channel. M-Switch SMTP may run many instances of each of these channels.

Inbound SMTP is handled by a single multi-threaded SMTP Server process (channel), so that new SMTP connections do not require new processes.

Message submission uses the SMTP protocol. The Internet Standard for Message Submission (RFC 6409) gives a special port for this (587), distinct from the SMTP port used for message transfer (25). Message submission is frequently authenticated, whereas this is unusual for message transfer. The SMTP Server can be configured to listen on both ports, or two SMTP servers can be used (one for each port).

Message Submission Library

The Message Submission Library provides a standard internal service for putting messages onto the message queue, which ensures that per message functions such as authentication, routing, and processing calculations are applied consistently and correctly. In older versions of M-Switch, this functionality was provided in a separate process. Key functions provided by the submission library are:

  • Sender validation.
  • Recipient validation and message routing.
  • Determining what internal processing is needed.
  • Policy and authorization checks.
  • Writing the message to disk.

The Message Submission Library is used in many components of M-Switch, and in particular in the SMTP Server.

M-Switch Configuration

M-Switch SMTP generally uses directory configuration, which has many advantages. Table based configuration is also provided, as it is useful for some specialized deployments.

The M-Switch GUI configuration is included in MConsole, which is described in the Messaging Configuration Management page.

M-Switch configuration is used by all of the M-Switch processes. Logically, configuration information from the directory is used by all processes. As changes to configuration are generally infrequent, this is implemented by QMGR obtaining configuration information from the directory using LDAP, checking for updates, and sharing it with the other processes.

Self Test

In order to optimize performance, the number of channels running, number of each channel type running, number of connections to a single MTA, will vary dependent on many factors, including remote system, network and local hardware performance. Manual optimization is complex, and in any event the optimal configuration is likely to vary over time. For this reason, M-Switch does a sophisticated “self test”, to measure performance and adapts configuration accordingly. This enables M-Switch to adapt its operational parameters to different hardware and network environments.

Message Routing

Routing of messages to other SMTP servers uses the domain part of the email address and is generally handled using the DNS (Domain Name System). M-Switch also allows for connections to be configured using MConsole to specific SMTP hosts. Reasons for doing this include:

  • Internal routing that over-rides DNS based routing.
  • Configuring security options, such as TLS for data confidentiality and SASL for peer authentication using passwords or strong authentication.
  • Configuring one or more permanent SMTP connections, which will reduce message switching latency by removing the time needed to establish a connection.

M-Switch also routes based on the whole email address. This is referred to as LASER routing, after the Internet working group that developed a specification with this name that did not get published as a standard. M-Switch broadly follows that specification, with a number of extensions, and options to deal with directory servers that store information in a different way.

The key concept in LASER routing is that you search in an LDAP directory for a mailbox of the form joe.user@mycorp.com. Routing succeeds when an entry is correctly matched. Additional attributes in the entry can control with channel and host the message is then sent to. Typically, the match will use a default LMTP connection to a local M-Box server.

Three basic ways that LASER routing may be used in conjunction with domain based routing:

  1. Single domain system, such as a consumer ISP or enterprise. If the domain is local, LASER routing is used, otherwise domain routing is used.
  2. Hosted service, handling multiple domains on one server. LASER routing is used to a single directory server. If there is a match, the message is handled according to the match. If not, domain routing is used for authorized message submission.
  3. Boundary switching to multiple departmental servers. Domain routing looks to identify locally configured domains. A match will identify the departmental LDAP server against which to do the LASER lookup, and the SMTP server to which matched addresses should be routed. This approach allows the boundary switch to fully validate email addresses by making use of existing LDAP servers.

M-Switch has flexible configuration that can support these setups and others. In addition to the core LASER routing, M-Switch also provides:

  • A mechanism to specify alternate mailbox addresses.
  • A list channel, that expands managed distribution lists.
  • Sub-addressing support (joe+XXX@myorg.com).

Event Handling

M-Switch uses the Isode event handling system. Multiple event streams can be configured, with event access using Isode’s Event View, Unix Syslog, or Windows Event Manager. Isode events are documented with description and severity.

More details on the event system are given in this whitepaper on operational monitoring.

Audit Logging & Audit Database

Detailed information on messages handled and processing status are recorded in an audit log. This includes full details of messages, Delivery Status Notifications, and Message Disposition Notifications. This audit log is processed by the Message Audit Service which is stored in an audit database. The audit database may be local or remote and used by one or more M-Switch servers.

The Audit Database is accessed by MConsole and the Message Operator Interface (MOI) which use this information to provide:

  • Statistics and reports on M-Switch usage and performance.
  • Message Tracking.
  • Message Archive Retrieval.
  • Quarantine management.

Further details are given in Message Operator Interface Overview.

Anti-Spam and Anti-Virus Channels

Detailed information on anti-spam and anti-virus are given under M-Switch Anti-Spam and M-Switch Anti-Virus. This section summarizes how these capabilities are provided in M-Switch. A number of anti-spam measures are implemented in the SMTP Server. Although messages may be rejected at the SMTP level, the SMTP server usually simply records the results of anti-spam checks in the message, so that they can be used in conjunction with other checks done in the anti-spam channel.

The core anti-spam function is performed by the Anti-Spam channel, using Isode developed technology. Isode spam checking does not use Spamassassin or other third party anti-spam technology. The anti-spam channel records its actions in audit logs, and may delete messages or move them from the message queue to the quarantine.

Users and operators may find information about messages in the quarantine using the MOI, or by email quarantine summaries sent out by the Quarantine Email Generator process, based on information in the audit database. A user or operator can use MOI to mark a message in the audit database to be released from quarantine. The Quarantine Release Process monitors for messages to be released, and does this by re-submitting the quarantined message.
The Anti-Virus channel operates in a similar manner, except that checks are performed using a third party AV checker. It is straightforward to use most AV checkers in conjunction with M-Switch.

Message Archive

Messages may be archived at the same time as they are placed into the M-Switch message queue. Messages are archived in a single file per message, in a file system directory. The archive directory is named using date/time, and changes at configurable intervals. This enables the archive directory to remain a reasonable size and to facilitate offline backup. The archive file name is audit logged along with other standard message audit information.

Message Tracking from MOI or MConsole can use information in the audit database to locate messages, and then use the SOM protocol to retrieve archived messages via the QMGR (which directly accesses the identified file in the archive), and display or forward them. Similarly, these tools can also view messages in the queue and quarantine.

Message Delivery Options

As well as standard delivery to M-Box using LMTP, M-Switch supports a number of other message delivery options. Messages may be delivered using a Local Channel to local mail files, either in standard UNIX formats or using the popular maildir format. Delivery time processing includes a delivery processing mechanism, controlled by a special language which defines the processing based on a wide range of message parameters. This can be used for personal functions such as automatically filing messages on delivery or sending holiday notifications. It is also useful to support system delivery functions. Modern systems are more likely to use LMTP delivery to a message store. Isode M-Box supports sophisticated delivery processing using SIEVE.

The Program Channel (or Shell Channel) enables delivery of messages to arbitrary programs. This can be used to easily integrate message delivery with special purpose delivery functions. This gives a general purpose mechanism for building system specific delivery functions and automatic mailboxes (e.g., 'echo' or 'delete').

Housekeeper Channel

A number of general functions are handled by a special Housekeeper Channel, including:

  • Processing Delayed Messages. During message submission, DNS and LDAP are used to determine routing and processing of messages. In the event of a timeout on one of these services, the message is “delayed” and message submission is completed later by the Housekeeper Channel.
  • Message Warning and Timeout. When a message has been in the queue for a configurable period, it will be timed out, and a delivery status notification sent to the sender. At intermediate times, messages warning of delay may also be sent.
  • Operator requested Message Deletion and Non-Delivery. In normal processing, message deletion is handled by the last channel doing work on the message. The operator may request message deletion or non-delivery using MConsole, which will be done by the Housekeeper Channel.
  • Message Redirect. The operator may request redirection of a message recipient to a new recipient using MConsole, which is done by Housekeeper Channel.
  • Message Reprocessing. Routing configuration for M-Switch may change in a way that would cause different routing for a message. The operator may request that a message or groups of messages are “reprocessed”, so that the updated routing calculation is applied. This is particularly important if a routing change is made to enable queue messages to take an alternate route.

Header and Message Content Processing

Internet messages are structured according to the MIME (Multipurpose Internet Message Extensions) Standard defined in RFC 2045. M-Switch has a MIMEShaper channel that understands MIME format and can perform conversion on messages and components within messages.

Individual body parts can be converted from one to another using conversion filters. Typically this is used for converting a text body part from one character set to another. The necessary conversions are calculated when a message is first submitted and they may be re-evaluated when a message is 'exploded'.

It is often desirable to rewrite header information in particular to 'normalize' addresses by rewriting the address in some canonical form, rather than one of the multiple addresses that can be used to reach a specific recipient. MIMEShaper provides options for the header normalization of Internet message headers are provided. This capability can be used to provide a coherent view of addresses for local users, or to manage addresses to give an external view in a firewall type configuration.

External Content Conversion

M-Switch provides the capability to use customer or partner provided capabilities to check and convert Internet messages, in addition to built-in capabilities.  This uses a protocol called CCCP (Content Checking and Conversion Protocol). The CCCP specification is available here.

M-Switch uses CCCP to provide information to a customer developed CCCP server, that performs checking and conversion of messages. CCCP is a simple text encoded protocol with an encoding approach aligned with the IMAP protocol. CCCP enables the following functionality to be provided:

  • Recipient changing (can be regarded as re-writing or redirect).
  • Sender re-write.
  • Recipient expansion (so the CCCP server can implement simple lists).
  • Content conversion. The CCCP server may provide back modified messages, and different copies can be assigned to different recipients.
  • For each recipient, message can be non-delivered, discarded, or quarantined (the CCCP requests M-Switch to perform the action).

These features allow a CCCP server to perform a wide range of address and content checking and conversion functions. The client/server architecture gives a number of advantages:

  • CCCP processing is applied to stored messages, so this is completely transparent to external message transfer.
  • Parallel processing means that slow CCCP processing of one message (e.g., taking human intervention) will not delay other message processing.
  • Conversion and Checking is cleanly decoupled from M-Switch.
  • For large deployments,  M-Switch and CCCP servers can be independently horizontally scaled.

M-Switch provides flexible use of CCCP, so that multiple CCCP channels can be configured to provide different conversions for different message flows. CCCP enables a wide range of conversion capabilities for different markets, and Isode is happy to work with partners to develop a new CCCP server

S/MIME

M-Switch can check S/MIME signatures on message submission. These checks are integrated with the authorization system, so messages can be controlled based on signature presence and validity.

M-Switch content conversion can strip S/MIME signatures, and can add an S/MIME signature. So M-Switch can be used to add S/MIME signatures at a boundary, or to add signatures where clients cannot do this.

S/MIME Encryption is supported by M-Switch Encryption, which is a capability that may be added to M-Switch

Security Labels

M-Switch provides support for Security Labels in a number of formats:

  • ESS Labels using the standard RFC 2634 encoding in S/MIME.
  • ESS Labels, base64 encoded in a configurable X- Header
  • Text encoded labels, represented in X-Header, Subject for First Line of Text (FLOT).

M-Switch can convert between label formats, using a SPIF based approach to map from ESS labels to text labels, and a regular expression approach to map from text format labels.

M-Switch can make authorization decisions based on presence and validity of a security label. It can also make Access Control decisions (checking against Security Clearance in the context of a governing Security Policy) based on destination channel, MTA, and user.

For more information see [Security Label Capabilities in M-Switch].

DKIM

M-Switch provides DKIM (DomainKeys Identified Mail) signing of messages, to verify the originating domain and message integrity. This provides a digital signature across message content and selected message headers to provide secure reputation support, which can be used to help protect against phishing attacks and spam.

Message Priority

M-Switch handles messages based on priority, which is available for all protocols (not just X.400). Message priorities for Internet messages are determined from MIXER "Priority: "Headers (three level X.400 priority) and from MMHS-Primary-Precedence (six level military priority) specified in RFC 6477. The List Channel may downgrade messages to a "bulk" priority, and Anti Spam channel may downgrade messages to "junk" priority. M-Switch has a number of capabilities for handling messages of different priorities, described in more detail on the M-Switch X.400 product page.

M-Switch can also be configured to map between various Priority fields and Importance.

Authorization & Routing Policy

M-Switch provides a comprehensive rule based authorization mechanism, for controlling which messages can be sent and how they are sent. Capabilities include:

  • Control based on sender and recipient, including address pattern matching.
  • Prevention of message relay.
  • Controls based on destination channel, which enables control of protocols and options used.
  • Control of content conversion.
  • Controls based on message size and message priority.
  • Controls based on S/MIME signatures and security labels.
  • Control for archiving.
  • Controls based on submitting IP address and sub-network and on submitting host and authentication. This is useful to separate out “internal” users, so that different policy can be applied. This is set up as “smtp-internal” and “smtp-external” in default M-Switch configurations.

Boundary Acknowledgement Support

M-Switch can be configured to generate acknowledgements for messages being relayed both read receipts (Message Disposition Notifications (MDNs)) and Delivery Status Notifications (DSNs). This is to support end systems which do not generate acknowledgements and security boundaries (e.g., Data Diode) where acknowledgements cannot be returned. There is an option to configure acknowledgement generation where they are not requested, and to control relayed acknowledgement requests.

Security and Authentication

Isode uses Transport Layer Security (TLS) for data confidentiality and Simple Authentication and Security Layer (SASL) for authentication. SASL is also used to map simple identifiers onto directory names for authentication. X.509 (PKI) based authentication can be used, by using TLS and SASL-EXTERNAL authentication. SASL and TLS are supported for SMTP Message Submission.

SASL and TLS are used by Isode’s SOM protocols, so that secure management from MConsole is possible.

Latency

M-Switch has been designed to give low transfer latency as well as high message throughput.

Internet Mail Conformance

RFC 1123 Requirements for Internet hosts - application and support, R. Braden, October 1989
RFC 1870 SMTP Service Extension for Message size Declaration, J. Klensin, N. Freed, K. Moore, November 1995
RFC 2920 SMTP Service Extension for Command Pipelining, N. Freed, September 2000
RFC 2033 Local Mail Transfer Protocol, J. Meyers, October 1996
RFC 2034 SMTP Service Extension for Returning Enhanced Error Codes, N Freed, October1996
RFC 2045 Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies, N. Freed, N. Borenstein, November 1996
RFC 2046 Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types, N. Freed, N. Borenstein, November 1996
RFC 2047 (Multipurpose Internet Mail Extensions) MIME Part Three: Header Extensions for Non-ASCII Text, K. Moore, November 1996
RFC 2049 (Multipurpose Internet Mail Extensions) MIME Part Five: Conformance Criteria and Examples, N. Freed, N. Borenstein, November 1996
RFC 2505 Anti-Spam Recommendations for SMTP MTAs, G. Lindberg, February 1999
RFC 2634 Enhanced Security Services for S/MIME, P. Hoffman, June 1999

RFC 3030

 SMTP Service Extensions for Transmission of Large and Binary MIME Messages, G. Vaudreuil, December 2000

RFC 3207

 SMTP Service Extension for Secure SMTP over Transport Layer Security, P. Hoffman, Feb 2002
RFC 3461 Simple Mail Transfer Protocol (SMTP) Service Extension for Delivery Status Notifications (DSNs), K. Moore, January 2003
RFC 3462 The Multipart/Report Content Type for the Reporting of Mail System Administrative Messages, G. Vaudreuil, January 2003
RFC 3798 Message Disposition Notification, T. Hansen, Ed & G. Vaudreuil, Ed, May 2004
RFC 3848 ESMTP and LMTP Transmission Types Registration, C. Newman, July
2004
RFC 4468 Message Submission BURL Extension, C. Newman, May 2006
RFC 4954 SMTP Service Extension for Authentication, R. Siemborski, A. Melnikov, July 2007
RFC 5321 Simple Mail Transfer Protocol, J. Klensin, October 2008
RFC 5322 Internet Message Format, P. Resnick, Ed, October 2008

RFC 5652

 Cryptographic Message Syntax (CMS), R. Housley, September 2009
RFC 5751 Secure/Multipurpose Internet Mail Extensions (S/MIME) Version 3.1 Message Specification, B. Ramsdell, January 2010
RFC 6152 SMTP Service Extension for 8-bit MIME transport, J. Klensin, N. Freed, M. Rose, E. Stefferud, D. Crocker, March 2011
RFC 6376 DomainKeys Identified Mail (DKIM) Signatures, D. Crocker, T. Hansen, M. Kucherawy, September 2011
RFC 6409 Message Submission for Mail, R. Gellens & J. Klensin, November 2011
RFC 6477 Registration of Military Message Handling System (MMHS) Header Fields for Use in Internet Mail. A. Melnikov & G. Lunt, January 2012