Spamassassin Benchmark

Purpose

Spamassassin is a popular open source product used for dealing with spam. This white paper sets out results of Isode's comparative benchmarking of Spamassassin and Isode's M-Switch Anti-Spam product. We have worked hard to perform evaluations in a neutral manner, and explain how we obtained our results. We encourage others to repeat our measurements.

Commercial Conclusions

This section of the white paper summarizes the conclusions that Isode has made from the measurements of this white paper. It is our intention to restrict the competitive analysis to this section, and that the rest of the paper simply sets out our observations and measurements on the two products.

Spamassassin is free software, whereas M-Switch Anti-Spam is a commercial product. However, in many cases, the TCO of Spamassassin will be higher because:

M-Switch Anti-Spam is considerably faster than Spamassassin. This means that in order to operate a service processing a given volume of mail, more hardware will be required to run Spamassassin, thus increasing the overall cost.
A higher quantity of (expensive) technical effort will be required in order to configure and operate Spamassassin, and to provide local support for a product without a vendor.

As well as the economic arguments, M-Switch Anti-Spam provides a better solution and in particular:

M-Switch Anti-Spam is significantly better at removing spam, as shown by the measurements of this report.
M-Switch Anti-Spam provides better management options, additional spam removal capabilities such as Greylisting, and more flexibility for the service operator.

Why Spamassassin

Isode has chosen to compare its product to Spamassassin for two reasons:

Spamassassin is well known and respected, and can quite reasonably be considered as a reference anti-spam system.
Spamassassin is quite popular with ISPs, who have high volume spam processing requirements.

Supporting Message Switch

M-Switch Anti-Spam includes SMTP message switching, and so can be deployed as a complete solution by a service provider. Spamassassin is a filter for detecting spam, and so needs to be used in conjunction with a message switch. It has been used in conjunction with a number of message switches.

Isode chose to use Postfix as the message switch to test in conjunction with Spamassassin. Postfix is one of the fastest free message switches available, and has an efficient integration of Spamassassin. We believe that this choice will give the most favorable performance numbers for Spamassassin. The most widely used message switch to support Spamassassin is probably Sendmail, using Milter to perform the integration. We believe that this combination would be significantly slower than using Postfix.

Performance Measurements

We performed our tests on a system with the following specifications:

Operating System	Red hat Enterprise Linux 3.0
CPU	2x1.4ghz PIII CPUs
Memory	2GB
Raid Controller	Adaptec 2100S 0+1 Raid 4x36gb 10k Write-Back
File System	Ext2 (noatime)

M-Switch Anti-Spam R10.2 was used with a standard configuration.

Spamassassin 2.63 (the latest released version) was used. A number of configuration changes were made to optimize Spamassassin performance, including:

Pairing it with Postfix as a local filter. Postfix essentially creates many parallel spamc processes which interact with the spamd daemon. This is important, as Spamassassin has a high startup cost.
Turning off DNS tests, which we found to cause significant Spamassassin delays, but with careful configuration did not significantly reduce throughput.

The throughput results are shown in the table below, measuring the number of messages submitted and delivered. Average message size is 10 kbytes.

Product	Performance
Spamassassin/Postfix	8 messages/second
M-Switch Anti-Spam	58 messages/second

The optimum Spamassassin performance shown here needed careful tuning with the test harness, as either providing a greater applied load or allowing the queue size to build up, caused significant performance degradation. At maximum load, the system was completely CPU bound, and dominated by operation of Spamassassin. Postfix, on the same platform, was able to switch around 100 messages/second, which makes clear that performance here is limited by Spamassassin.

At maximum performance, M-Switch Anti-Spam is disk i/o bound, and performance is primarily limited by the raw message switching performance, and not by the anti-spam analysis.

Spam Removal

A basic measure of an anti-spam product is how well it removes spam. We tested spam removal by using a collection of 1,650 messages, which are all recent messages taken from "spam trap" mailboxes (email accounts that receive only spam). The results are shown in the table below.

Product	SPAM	POSSIBLE SPAM	False Negatives (Spam missed)
Spamassassin	66%	n/a	44%
M-Switch Anti-Spam	81%	9%	5%

Spamassassin has the concept of a threshold, which we left at the default level of 5. M-Switch has a similar approach, but multiple levels (with different processing options) are possible. We used our recommended settings, which give categorizations of "SPAM" and "POSSIBLE SPAM". This reflects the reality that a higher level of spam detection will lead to a higher rate of false positives. DNS checking was enabled for these tests.

We performed the same tests on 1,000 spam messages, taken from about 6 months back. The results for this older spam are shown below.

Product	SPAM	POSSIBLE SPAM	False Negatives (Spam missed)
Spamassassin	96.5%	n/a	3.5%
M-Switch Anti-Spam	94%	2.5%	3.5%

It is striking how much more effective both products are on dealing with old spam. We believe that this reflects the fact that spam is getting harder to detect, which is an ongoing challenge for anti-spam vendors. The changes with more recent spam statistics suggest that M-Switch Anti-Spam has done a better job of providing than SpamAssassin.

False Positives

A key measure of an anti-spam solution is the level of false positives. It is reasonably straightforward to measure false negatives fairly, as spam is quite uniform and consistent (despite folk lore to the contrary). False positives depend significantly on the nature of the real traffic. We ran tests using two sets of data. The first data set was 1200 messages, selected from a wide range of sources to give coverage of a wide range of "ham" (messages that are not spam), including a selections of "hard ham" (real messages that are deemed to be difficult to separate from spam).

Product	False Positives
Spamassassin	0.9%
M-Switch Anti-Spam ("SPAM")	0.15%
M-Switch Anti-Spam ("SPAM" and "POSSIBLE SPAM")	1.15%

In most cases, both for Spamassassin and M-Switch Anti-Spam, it is clear why messages are marked as spam, and that these messages are usually of a type which are of relatively low interest to the end recipient.

This data also shows the strong benefits in separating out data as "SPAM" and "POSSIBLE SPAM". For most users, it is reasonable to assume that all messages in the "SPAM" category really are spam, and to occasionally review messages in the "POSSIBLE SPAM"

Functional Differences

Review of the functional differences between SpamAssassin and M-Switch Anti-Spam can be done independently, by review of the documentation on each product. Features offered by M-Switch Anti-Spam, that give management and deployment advantages over Spamassassin include:

Graphical user interfaces for configuring anti-spam capabilities.
Easy management configuration of detailed anti-spam behavior and word blocking.
Use of same product for spam detection, and content filtering based on word blocking.
Multi-level spam categorization.
Choice of anti-spam processing actions and behavior, which can be configured for each user, based on spam level.
Support for Greylisting, which is an additional effective technique for spam removal. This was not used as a part of this benchmark.

Isode Whitepapers

Stay Informed

From the Isode blog...

Creative Commons

Benchmark and Comparison of Spamassassin and M-Switch Anti-Spam