This page describes the content checking and processing capabilities of M-Switch SMTP. This page is split into four sections, reflecting groups of capability; General Purpose message conversion, Reputation services and M-Switch support of SPF and DKIM, Anti-Virus handling (using third party AV checkers) and Anti-SPAM Capabilities and Dirty Word checking.
General Purpose Message Conversion
Header and Message Content Processing
Internet messages are structured according to the MIME (Multipurpose Internet Message Extensions) Standard defined in RFC 2045. M-Switch has a mimeshaper channel that understands MIME format and can perform conversion on messages and components within messages.
Individual body parts can be converted from one to another using conversion filters. Typically this is used for converting a text body part from one character set to another. The necessary conversions are calculated when a message is first submitted and they may be re-evaluated when a message is 'exploded'.
It is often desirable to rewrite header information - in particular, to 'normalize' addresses by rewriting the address in some canonical form, rather than one of the multiple addresses that can be used to reach a specific recipient. Mimeshaper provides options for the normalization of Internet message headers. This capability can be used to provide a coherent view of addresses for local users, or to manage addresses to give an external view in a boundary messaging configuration.
External Content Conversion
M-Switch provides the capability to use customer or partner provided capabilities to check and convert Internet messages, in addition to built-in capabilities. This uses a protocol called CCCP (Content Checking and Conversion Protocol).
M-Switch uses CCCP to provide information to a customer developed CCCP server, which can then perform checking and conversion of messages. CCCP is a simple text encoded protocol with an encoding approach aligned with the IMAP protocol. This allows a CCCP server to provide the following functionality:
- Recipient changing (can be regarded as re-writing or redirect).
- Sender re-write.
- Recipient expansion (so the CCCP server can implement simple lists).
- Content conversion. The CCCP server may provide back modified messages, and different copies can be assigned to different recipients.
- For each recipient, message can be non-delivered, discarded, or quarantined (the CCCP requests M-Switch to perform the action).
These features allow a CCCP server to perform a wide range of address and content checking and conversion functions. The client/server architecture gives a number of advantages:
- CCCP processing is applied to stored messages, and so is completely transparent to external message transfer.
- Parallel processing means that slow CCCP processing of one message (e.g., taking human intervention) will not delay other message processing.
- Conversion and Checking is cleanly decoupled from M-Switch.
- For large deployments, M-Switch and CCCP servers can be independently horizontally scaled.
M-Switch provides flexible use of CCCP, so that multiple CCCP channels can be configured to provide different conversions for different message flows. CCCP enables a wide range of conversion capabilities for different markets, and Isode is happy to work with partners to develop a new CCCP server.
Reputation services provide a mechanism for organizations originating messages to provide information that enables message recipients to verify that the message comes from its claimed source. Ideally, all email should make use of reputation services. In practice, reputation services can help differentiate in anti-spam checks, by making reputation information available as input to the anti-spam checks. M-Switch supports two reputation services.
M-Switch provides DKIM (DomainKeys Identified Mail) signing of messages, to verify the originating domain and message integrity. This provides a digital signature across message content and selected message headers to provide secure reputation support, which can be used to help protect against phishing attacks and spam.
SPF (Sender Policy Framework) makes use of DNS (Domain Name Service) configured information. Setting up SPF is part of DNS configuration, independent of M-Switch. M-Switch can perform SPF checks on inbound messages. There are two approaches to handling:
- Reject messages at the SMTP server when SPF checks fail.
- Mark the message with a special header, which can be used in subsequent anti-spam checks.
Operationally, the second approach is usually more useful.
M-Switch provides anti-virus checks on some or all messages being handled, using third party anti-virus packages. The following anti-virus packages are supported:
- Sophos, a commercial product that is optimized for this type of checking. This can be purchased directly from Sophos.
- Norman, a commercial anti-virus product that works well for this type of checking. Isode resells Norman.
- ClamAV is an open source anti-virus checker, that is specifically designed to target email-borne viruses and malware.
Setup of M-Switch to use the anti-virus package of your choice is straightforward.
What does M-Switch do to support Anti-Virus checking?
The basic function of M-Switch to handle viruses is very simple. It takes an inbound stream of SMTP messages, separates out the message content to hand to a virus checker, and then sends the messages onward by SMTP (once they have passed the virus check). M-Switch can be easily inserted into an SMTP message stream, to add anti-virus capability. The more detailed process is:
- M-Switch has the concept of "channels" which perform specific functions on messages in the internal queue. A content checking channel drives the anti-virus capabilities which M-Switch uses. This is programmable, so different content checking channels may be invoked (by the same instance of M-Switch) with different parameters in different situation, or even with different virus checkers.
- M-Switch can be configured to invoke the anti-virus checking on all messages, or on selected messages (e.g., "all inbound", "all outbound", "all messages from organization X", "all messages to user X").
- M-Switch can control virus checking by size. In particular, virus checking can be skipped for very small messages (which are common and will be too small to carry a virus).
- The virus checking can do various things on detecting a virus, including one or more of:
- sending a customizable message back to the sender
- sending a customizable message on to the intended recipient (example below)
- removing the infected body part, and then replacing it with another body part (typically one that says "there was a virus infected thing here")
- if the virus checker can clean up the virus, the channel can replace the infected body part with a clean one
- The virus checking audit logs all activity, which can be processed into management reports as needed.
- Anti-virus statistics can be displayed in MConsole, when the audit database is used.
- The virus channel has a framework which can be used with any virus checker that provides an API or command line interface. Integration is straightforward. While the virus checker is usually run on the same machine as M-Switch, it can also be set up to run remotely.
M-Switch provides a range of anti-spam capabilities using Isode developed technology (not third party). When a message is determined to be “spam”, M-Switch may delete messages or move them from the message queue to the quarantine.
Operators can access the quarantine using MConsole, which enables them to search for messages and release from quarantine.
Users will be sent regular "quarantine" email messages, containing URLs which enable (after authentication) easy release or viewing of quarantined messages. Users can also access this web interface to search their quarantined messages, and can manage personal white lists, email schedule, and other anti-spam preferences.
Dirty Word Checking
M-Switch’s anti-spam engine can also be configured to perform dirty word checking - for example to block messages based on profane works, or on sensitive words (e.g., security references) that are not allowed. This can be used for inbound or outbound checks.
Catching Spam with M-Switch Anti-Spam
M-Switch Anti-Spam uses a number of different, complementary techniques to determine the 'Spam Score' of a message. You can use any or all of these techniques. Currently supported techniques include content filtering, grey listing, phone & URL blacklists, subject line matching, originator matching, host matching, real time black hole lists, message characteristic checking, network address checking, obfuscation techniques and trigraph checking.
The Audit Database shipped as part of M-Switch stores structured audit log data from one or more copies of M-Switch in an ODBC compliant database. That data can be accessed by Isode applications or third-party applications supplied by Isode Partners.
Isode applications shipped with the current release of M-Switch Anti-Spam include message tracking, message quarantine and comprehensive statistical reporting, all functions delivered via the Message Operator Interface. Operators can access the same functionality with MConsole.
Quarantine management includes a facility to send HTML messages to users with a list of messages in quarantine. This is provided as a script, which can be customized for each installation.
This message provides a listing of the messages, and a URL which causes the message to be released from quarantine. Release works by updating the status in the audit database to "pending release". A background process releases messages from the quarantine and marks status to "released".
Looking at the content of a message to match words, phrases and other information is one of the most effective ways of eliminating spam. Our content filtering measures work by using a technique known as Support Vector Machines, a significant improvement over the Bayesian logic used by most other anti-spam vendors.
In Bayesian analysis a sample set of messages and spam are analyzed, and each input (word or spam feature) being checked is counted, with weight assigned to each input. Support Vector Machines generate weights by looking at the inputs in combination, taking into account the relationship of the inputs and how they occur in spam, rather than treating each input in isolation.
Grey listing works by recording send, recipient and source IP address, letting through known tuples, temporarily failing all other messages (forcing legitimate sending systems to retry sending the message) and adding as a known tuple when a retry is received.
As most spam is sent by scripts which do not retry failed deliveries, properly implemented grey listing can remove a high percentage of spam before it gets to the Message Transfer Agent (MTA). A downside of grey listing is that it impacts a small percentage of real message traffic.
Phone & URL Blacklists
Whilst most spammers will fake return addresses, they nearly always include in the body of the message at least one method (phone or website URL) so the recipient can respond to the spam's advertising. M-Switch Anti-Spam maintains both phone and URL blacklists.
Prevention of Relay
The Authorization setup has a recommended default setup, which prevents relay of non-local messages, which is important to prevent spam.
English Trigraph Checking
Looks for and scores the frequency of text strings which contain three-letter combinations that do not exist in the English language.
The SMTP channel has a number of stringency checks, such as not accepting messages from IP addresses which do not resolve to domain names, which will help to reduce spam.
Address Harvesting Prevention
After invalid addresses are received, the SMTP channel will slow down. This is believed to be the most effective way to guard against address harvesting. SMTP commands selection is configurable. In particular, the VRFY command should be configured off for external SMTP.
Other Anti-Spam Techniques
- Subject Line Matching: Matching the subject line against a list of topics that should always be treated as spam.
- Originator Matching: Matching the originator of the message against an email blacklist.
- Host Matching: Matching the sending host against a host blacklist.
- Message Characteristic Checking: Checking the technical characteristics of a message, such as the way in which returned messages are handled.
- Network Address Checking: Checking the originating network address.
- Obfuscation: Checking for spam obfuscation techniques such as HTML comments or messages that are composed entirely of URLs.