May 16th, 2008
Posted in Uncategorized
Real World Scenarios
Despite all the money invested into anti-spam solutions, spam volume continues to rise. Yes, spamming is an arms race. But the real race is one of sheer volume.
Spammers respond to difficulty by simply sending more spam. Better filtering? Send more to improve numbers getting through. Spamming not profitable enough? Send more spam. Users not interested? Send more variety. With botnets, spammers have a highly scalable delivery infrastructure and are not limited by resources. Unfortunately, it’s the receiver of spam that bears the cost of that volume.
The problem is more than just the annoyance of spam. Spam is a big cost to organizations. High spam volumes lead to delays in email delivery and significant over-capacity to handle spikes in volume. Email providers know customers are very sensitive to any delays in the receipt of important email, and any service disruptions by a failure to handle loads can have immediate complaints and ongoing financial impacts.
Delays in email delivery caused by high spam traffic divert IT attention to chase spam.
Ongoing IT workload costs likely dwarf one-time capital expenditures for new systems.
Adding capacity in chunks with each budget period makes it difficult to know if it’s too little or too much to scale capacity to meet volumes.
Traffic shaping reduce IT infrastructure and support costs because it removes more spam at the connection level than any other approach.
One of the Fortune 500 companies MailChannels works with has implemented traffic shaping solely to get their infrastructure costs under control. They were being flooded with spam and as a result legitimate email was being crowded out by the spam resulting in delivert delays of hours at a time. Their spam filters were getting rid of it so the end users didn’t see it but the servers were doing all they could to process backlogged traffic. The company couldn’t accept any more mail, they were are there limit in terms of concurrent SMTP connections and were at a loss to come up with a good strategy for dealing with all the spam.
They were using all the blacklists they could find, but even though the blacklists got rid of 50 to 70 percent of spam coming from known spam sources, the spam that got through was significant enough to be a very serious problem for end users and administrators trying to keep the email service flowing.
Implementing email traffic shaping in front of their servers dramatically dropped spam from 70 percent of all processed traffic down to 20 percent overnight as a result they turned off 4 of the 6 servers they were using to handle all inbound mail. More importantly, they no longer needed to waste time maintaining content filters, adding more servers or experiencing slow SMTP responses.
There are limitations with every anti-spam technology. While filtering is an effective at separating spam from email, it is only one layer in a multi-tiered anti-spam architecture designed to leverage various technologies suited to each task. Applying traffic shaping at the network edge ensures legitimate senders get excellent quality of service and their mail flows quickly, while spammers are given very poor quality of service and their mail is not allowed into your network.
NEXT: Post #10 Challenges of Traffic Shaping
PREVIOUS: Post #8 Dealing Spammers a Blow
Tags: cost, high traffic loads, itunes, smtp, spam, traffic shaping
April 30th, 2008
Posted in Uncategorized
Slowing Things Down
The problem is, typical email systems work in a queue. This means that high spam traffic clogs your network and crowds out legitimate mail. Botnets pour messages into your network, and mail servers receive the messages as quickly as they can. Next, the spam filter analyzes and tries to filter out any messages that appear to be spam.
Filters are effective at separating spam from email but do nothing to stop the rising volume of SMTP connections hammering the server. When spam traffic rises, the server becomes overloaded and results in delivery delays for all email, similar to how a backlogged exit ramp can impede the flow of traffic on a highway during peak hours.
Today, Internet facing email servers accept thousands of emails per minute. As spam volume increases, so too does the CPU required to process all that mail. The blunt solution is to scale hardware to keep up with volume but this is a one-to-one cost — the more volume, the more servers are needed.
The fact is spam filters aren’t getting a whole lot more accurate, and it certainly doesn’t help that blocking spam is a reactive approach — a sender needs to be identified first before rules or signatures are updated. Filters will always be playing catch up with the spammers.
If you block based on reputation, what do you do when a new spam campaign breaks out and the sender has never been seen before?
What is needed is a way to get rid of the spam and prioritize legitimate mail without having to receive all the messages first or know who the bad senders are before hand.
To use the highway analogy, what if you could put good senders in an express lane and the spammers in the slow lane so that legitimate email can be delivered first?
NEXT: Post #8 Dealing a Blow to Spammers
PREVIOUS: Post #6 Blocking Spam in 2008
Tags: anti-spam, content-filters, high traffic loads, spam, throttling, traffic shaping, trickle blog
April 7th, 2008
Posted in Uncategorized
Once Promising Proposals for a Final Ultimate Solution to the Spam Problem (FUSSP)
“Two years from now, spam will be solved.”
That was Bill Gates’ famous pronouncement back in 2004. Microsoft, Yahoo and the open source community devised two techniques that they believed would eradicate spam. The first was sender authentication, which allowed email senders to provide a list of the servers permitted to send email for users within their domain. The idea was that sender authentication would eliminate spammers spoofing legitimate email addresses, and allow for the creation of a permanent, ironclad white list of trustworthy domains that never send spam, thus allowing recipients to simply block everything not on the white list and end spam forever.
Another idea pitched in 2004 was the computational challenge. Senders would, upon connecting to a receiving email server, have to spend considerable CPU cycles computing the answer to a mathematical challenge provided by the receiving server. Bill Gates believed this approach would stop spam by making it cost too much to send the high volumes of email required to make spamming profitable.
Unfortunately, neither sender authentication nor the computational challenge technique resolved the spam problem. Computational challenges were rejected as being too costly for legitimate bulk email senders (airlines, banks, open source mailing lists, etc.) And sender authentication while eventually enjoying wide-spread adoption in the form of DKIM and SenderID, proved prone to errors. As as result it has remained useful mostly for the acceptance of legitimate email and phishing protection rather than the rejection of spam.
By 2005, what the anti-spam community was getting right was content filtering. When spam filters had reached above the 90 per cent accuracy level, spam transitioned from a problem of content to a problem of volume, the spammers simply send more spam. And they can do this because the recipient pays the cost of content filtering rather than the spammer.
The cost of a resource-consuming filtering system increases during high traffic loads. If you block spam content, spammers will find new ways to get around it. Bill Gates was right, the only way to stop them is to create difficulty by making spam too costly to send. If you do spammers are left to find new targets that are easier to hit.
NEXT: Post #4 Spamonomics: The Economics of Spamming
PREVIOUS: Post #2 Prohibition Induces “Botlegging”
Tags: accuracy, anti-spam, bill gates, content-filters, dkim, economics, high traffic loads, microsoft, spam, spammers, spamonomics, yahoo
March 28th, 2008
Posted in Uncategorized
A Short History of Spam Protection
While methods have changed, spam continues to be the misuse of an open communication network for financial gain. What was once a harmless annoyance has led to serious conditions where high spam traffic can clog email servers to the detriment of legitimate mail.
How did we get here? And what can we change to solve the problem?
The first spam email ever was used to promote a seminar from Digital Equipment Corporation (DEC) in 1978. I’d call it spam because it was a mass emailing harvested from a printed directory of ARPAnet to recipients who had not requested any contact.
Spam didn’t become a huge problem until around 2002 when there were enough active email users worldwide to make spamming profitable. In response, the first commercial and open source spam filters arrived in Brightmail, PureMessage, and SpamAssassin to name a few. The first generation of filters applied sets of rules to each message received, identifying features within messages which might indicate the likelihood
of being spam.
Spammers countered rule-based filters by obfuscating the content of their messages. Rather than sending a text message advertising Viagra, for example, the spammer might chop the message into small HTML pieces which, while unrecognizable to the spam filter, would still render into legible text for the message recipient. The rule-based filters added more rules to catch these obfuscations, causing the spammers to further innovate. This pattern of content obfuscation continues to the present day, the most recent example of which is probably MP3 spam (i.e. spam message contained in an audio file).
Anti-spam is one of those areas of IT where you’re “damned if you don’t.” If email is flowing free of spam, you hear nothing. But when spam is getting through or emails are backlogged on the server, there’s hell to pay.
Why is spam causing backlogs? Why is all mail treated equally? And do we need to keep adding what are effectively junk processing servers?
As the sophistication of spam has increased so has the need for processing power to analyze those messages. Today, with email servers under high traffic loads, the ever increasing computational cost and processing overhead of analyzing the content of every email often results in service disruptions for legitimate email. This has to change. IT infrastructure costs should be a function of legitimate activity not spammer driven loads.
To solve the loading problem imposed by the current method of spam filtering where all incoming email messages are accepted by the server, buffered in a common queue on a first-come first-served basis, there needs to be a shift away from a single-queue of email traffic towards a prioritized system that can expedite legitimate mail first.
But there’s more that needs to be considered…
UPDATE: On the subject of the history of spam, Christopher Nickson writes that the word “spam” to describe unsolicited commercial email recently celebrated it’s 15th anniversary.
NEXT: Post #2 Prohibition Induces “Botlegging”
Tags: anti-spam, backlogs, email, high traffic loads, itunes, smtp, spam, trickle blog
