I recently grew frustrated at the lack of a good, fast RBL checking web site. So, during the MailChannels summer hackathon on a remote island in the Pacific Northwest, I coded up an AJAX-based blacklist check web page. This new service is a work in progress, but already satisfies a few goals I had in mind at the start of the hackathon:
It’s fast – DNS lookups are done in parallel, so a complete set of results usually comes back in under one second;
It’s easy – IP addresses and host names are accepted, and CIDRs will be added soon; and,
It gives you useful information – such as the ASN and subnet range from which the IP address was allocated.
For those of you who are so inclined, feel free to figure out the wire-line protocol, which is a REST-ful JSON thing. I can’t promise it won’t be rate limited, but I certainly don’t mind if you script it to provide the back-end for your own multi-RBL checking service.
MailChannels protects your IP address reputation by analyzing outgoing email traffic and automatically applying policies on users to prevent internal abuse. MailChannels boasts three key advantages are:
Transparent Filtering – analyzes outgoing email traffic, and applies policies without requiring major configuration changes.
High Scalability – MailChannels handles 30 million messages per hour, and greater than 10,000 connections per second.
Blacklist Avoidance - senders are re-mapped to new IP addresses as soon as they start behaving suspiciously. This reduces the impact on your network’s IP reputation.
“Outbound spam filtering is all about ensuring reliable email delivery. If your organization counts on email delivery, then you should invest in outbound spam filtering.”
We’ve written a new white paper that discusses the need for a multi-layered approach in dealing with outbound email abuse (i.e. outbound antispam). The layers are thus:
Accurate content filtering – using a great spam filter to tag and potentially reject spam messages before they leave the network;
Local reputation management and policy enforcement – keeping track of the reputation of “senders” in your network, and preventing “bad” users from getting too abusive; and,
IP address management – moving traffic out through different blocks of IP addresses depending on the reputation of the sender.
The idea of rolling out these techniques is to do on the sending side what receivers are doing on the receiving side – except with the benefit of knowing more about your senders, such as their “account id” or “phone number” depending on what kind of network or service you’re operating. We’d love to hear your feedback on this new white paper, so go ahead and download A Multi-Layered Approach to Effective Outbound Spam Protection and let us know what you think.
In 2004, Bill Gates announced that the global spam problem would be completely eliminated in two years as a result of new technology from Microsoft. The new technology he was talking about was basically a way to make email senders pay a small fee for each email message they send. Spammers would not want to pay exorbitant amounts to send email, and would therefore stop spamming the world. Many of us in the anti-spam community laughed at Bill’s suggestion, because it seemed ridiculous that everyone in the world would ever “universally” participate in such a scheme.
But, in a way, I think Bill Gates’ vision has actually come to fruition. If you’re trying to send marketing email or transactional email, you will very quickly discover that it costs money to do so effectively. And now, if you use Amazon’s cloud services for your hosting needs, you’ll find it costs real money to send email from there as well – at least, if you want to send it reliably.
Amazon’s Problem
Amazon Web Services, LLC, the subsidiary of Amazon.com that operates the formidable Electric Elastic Compute Cloud (EC2), announced yesterday the availability of a new email sending service aimed at users of EC2 and Amazon’s other cloud services. In my opinion, the new service, which is called (predictably) “Amazon Simple Email Service,” aims to resolve a longstanding problem with EC2 – namely, that the spam problem in EC2 had ruined EC2′s IP reputation and made it next to impossible to send email from Amazon’s cloud.
The problem started in late 2009, when Amazon EC2 users started complaining in the support forums that their outbound email was being blocked worldwide by the Trend Micro MAPS blacklist service. Despite many appeals to Trend Micro, the blocking continued for a considerable time, causing Amazon EC2 users a great deal of “pain”. Other blacklists joined in, including SORBS; and Spamhaus added Amazon’s entire EC2 IP space to the Policy Block List (PBL). The blacklists had good reason to list EC2′s IP address space: Spammers had gone literally nuts on the Amazon service, signing up accounts and sending spam indiscriminately and in very, very high volumes. And Amazon, despite tasking a team to deal with the problem, was basically unable to convince the world that its dynamic cloud could ever be trusted.
In response, Amazon created Simple Email Service. And now, if you want to send email reliably from the Amazon cloud, all you need to do is sign up for an SES account and learn to use their email sending API. Amazon has built a reputation system to track the content (i.e. spam) and complaint histories for each SES customer, with sending volume gradually increasing as an SES customer demonstrates its ability to send good email that the world wants. Messages cost $0.10 per 2,000 messages delivered.
Our Analysis of Amazon Simple Email Service
Amazon Simple Email Service will do very well, financially. Almost all EC2 users need to send email reliably, and I think a good proportion of users will opt to send via the SES service. Competitors like SendGrid and authsmtp.com will feel some pain, since many of their customers are frustrated Amazon cloud users; however, the more innovative providers (like SendGrid) will prosper as a result of their much richer offering, servicing customers who need much more than just email delivery.
Spammers will abuse the service, of course. I fully predict that spammers will sign up for SES accounts, using credible “front” companies to obtain accounts, and patiently build up their sending volume until it is “ripe” for a bulk email blast. This will repeat and repeat endlessly, with offshore “mechanical turks” (another Amazon innovation) cranking out new SES accounts daily to feed the beast. At the end of the day, whether Amazon SES succeeds in providing truly reliable email delivery will hinge on how good their outbound spam filtering technology is, and how quickly they act when an SES account starts to go bad.
I’m guessing that Amazon will invest enough to do a good job, and that SES will emerge as one of the largest email senders in the world by the end of 2011. After all, they’re being paid to do a good job, which provides just the kind of incentive Bill Gates was talking about.
One of the questions we sometimes get from prospective customers is: how well does slowing down spammers actually work, versus other techniques like connection blocking? The answer: After a great deal of analysis, it works very well indeed.
To compare how well throttling (i.e., slowing down or “traffic shaping” – or sometimes “tar-pitting”) connections performs as an anti-spam measure we decided to compare it against the data we had on Spamhaus blocking effectiveness.
We analyzed roughly 66 days worth of data across several of our customer sites from 2010-02-01 to 2010-04-08.
Connections from 28,204,693 distinct IPs were analysed during this time. All of the connections analysed were checked against Spamhaus’ Zen RBL at the time of connection, using up-to-date Zen data. Of those that were not rejected by Spamhaus, throttling was enabled on many of these connections. Of those that were throttled, the client either aborted the connection (we will call this “rejected by throttling”, even though it is not technically correct), or the client remained connected long enough to properly complete the SMTP session. We did not look at any connections that were throttled if Spamhaus was not first queried.
We found that 24,812,576 of these IPs were always rejected by Spamhaus, implying that they must have either a) already been listed on Spamhaus prior to the time of our test (most likely) or b) detected immediately by Spamhaus (less likely).
1,947,976 IPs were rejected by throttling at some point. This means they were not rejected by Spamhaus, and so were throttled and disconnected due to being throttling.
951,637 IPs were never rejected by Spamhaus and also never rejected by throttling during this time-frame. Remember, all connections in our sample were either rejected by Spamhaus or subsequently throttled. Therefore, these unsuccessfully throttled connections were either legitimate senders who were throttled as a result of having a neutral or unknown reputation, or bad senders that were both not on Spamhaus during this time and yet able to handle throttling. 638,626 of these IPs were throttled less than 10 times, which indicates unknown IPs initially being throttled. 453,556 of these IPs were throttled no more than 4 times.
From the 1,947,976 IPs rejected by throttling 559,562 IPs were rejected by throttling first and then later rejected by Spamhaus. We will look at these IPs in more detail as these are the ones that will allow us to compare Spamhaus vs throttling.
These 559,562 IPs were rejected by throttling on average for 4 days (351,682 seconds) prior to being rejected by Spamhaus. This ranged from 1 second to almost the entire time-frame of this analysis of 66 days (5,771,484 seconds). Therefore it is not unreasonable to assume that a spambot could go undetected by Spamhaus for longer than this time frame.
These 559,562 IPs were rejected by throttling on average 30 times, ranging from once to 158,378 times per IP.
Once these 559,562 IPs were detected by Spamhaus, Spamhaus then rejected on average 116 times, ranging from once to 220,084 times per IP.
Disregarding throttling rejections, if we look at the 26,340,897 IPs that at some point were rejected by Spamhaus, Spamhaus rejected on average 30 times, ranging from once to 1,034,038 times per IP. This shows that the IPs that were first detected by throttling go on to be more actively rejected by Spamhaus : 116 times versus only 30 times for ones we did not first detect by throttling. Something to consider is that throttling is likely to see newer IPs, since Spamhaus has not recognised them yet. We can assume that newer IPs send more spam.
Since we only throttle reject on average for 4 days (see above) before Spamhaus takes over, our average of 30 rejections per IP is actually higher than Spamhaus’s average of 30 rejections per IP, since this is averaged over the entire time frame of 66 days.
Following is a graph showing the amount of time it took for Spamhaus to “catch up” to the IPs that we were able to reject successfully with throttling:
One of the things that we keep track of with Traffic Control is the percentage of SMTP connections which end without the sender issuing a QUIT command as required by RFC 5321. Because the QUIT command is required by the RFC, it’s possible to reject and prevent delivery of messages from senders who violate this requirement. The impact on legitimate senders is extremely low, because all known legitimate MTAs properly issue a QUIT.
For a very long time, the percentage of connections that did not issue QUIT hovered around 0.25%. But for some reason, in the past day this figure has shot up. The graph below details this development:
This graph shows a recent spike in the percentage of connections passing through Traffic Control servers which are not correctly issuing the QUIT command to end an SMTP connection.
The data is the same whether we look at individual customer sites, at inbound-only traffic, or at outbound traffic emanating from large ISP sites where we have installed Traffic Control for transparent outbound spam filtering. Any ideas as to why this is happening? Anyone else seeing this trend?
"In the history of cryptography, the bombe was an electromechanical device used by British cryptologists to help break German Enigma-machine-generated signals during World War II. The bombe was designed by Alan Turing, with an important refinement suggested by Gordon Welchman. The bombe was named after, and inspired by, a device that had been designed in 1938 by Polish Cipher Bureau cryptologist Marian Rejewski, and known as the "cryptologic bomb" (Polish: "bomba kryptologiczna")."
In my two previousposts, I wrote about how the Rustock botnet is now sending a great deal of spam via TLS-encrypted connections and the variety of problems this new trend in spamming activity is causing and will cause for the Internet community. I mused that Rustock is probably encrypting spam because this allows its traffic to remain un-inspected by network analysis tools that perform man-in-the-middle analysis of SMTP traffic. In my second post, I discussed some of the problems that encrypted spam is causing for major email receivers, including increased TLS encryption load.
In this post, I will cover off a few things that receiving and sending networks can do to reduce the impact and amount of TLS-encrypted spam traversing the Internet. Fortunately, this is one of those spam problems where it’s not only up to the receivers to fix the problem.
What can email senders and receivers do to stop TLS-encrypted spam?
The only way to stop spammers from using TLS encryption is for the receiving mail server to tell the spambot that it does not support encryption. To understand how this is accomplished, let’s first have a look at how the ESMTP protocol advertises SMTP extension mechanisms, one of which is the STARTTLS command that initiates an encrypted session.
Advertising is not always a good idea
When an SMTP client (i.e. the machine sending email) issues the EHLO command at the start of an ESMTP connection, the SMTP server responds with a list of extension commands it supports. The SMTP client can then issue any of the commands listed. A typical response might look like the following:
250-ESMTP Server Ready
250-SIZE 0
250-DSN
250-STARTTLS
250 TLS
The STARTTLS command listed above allows the SMTP client to issue the command “STARTTLS”, after which a bunch of random looking garbage representing the actual TLS encrypted session is exchanged back and forth across the TCP connection.
If the mail server does not list STARTTLS when the EHLO command is issued, then the SMTP client may not send the STARTTLS command. This effectively prevents encryption from happening.
Interesting side-note: Most of the major consumer-oriented email receivers of the world do not ever advertise STARTTLS. I tested Hotmail, Gmail, Yahoo!, and AOL and found that none of them support TLS. Microsoft’s Exchange Hosted Services (formerly Frontbridge) does advertise STARTTLS. So does Symantec’s MessageLabs service. These are both enterprise-oriented services whose customers are most certainly more demanding of privacy and security than the typical customer of a consumer-oriented email service like Hotmail.
Senders and receivers both play a part
Email receivers who are concerned about encryption load can either completely disable TLS-encryption, or can be selective about whom they offer it to. If your email receiving system has a reputation system attached to it, you can implement a policy whereby STARTTLS is only advertised to IP addresses that have established a positive reputation – everyone else has to send in the clear, leaving less decryption burden on your CPUs.
Sending networks (e.g. ISPs, hosting companies, university campuses… basically anyone with lots of machines that can send on port 25) can also filter STARTTLS, assuming they have the right technology in place in the network. I don’t usually trumpet MailChannels technology in this blog, but in this case I’m going to go out on a limb and suggest that sending networks consider using a <a href=”/product/outbound.html”>transparent SMTP proxy</a> such as Traffic Control in the outbound path of their port-25 traffic. We – and a few others – can selectively block the STARTTLS response when senders from within the sending network are trying to connect to mail servers to deliver their deadly encrypted payload.
Conclusions
Incidentally, it seems like Rustock has calmed down w.r.t. sending encrypted spam of late. In the last 24 hours, only a tiny fraction of the connections emanating from our outbound filtering customer networks have been encrypted. Have we seen the end of this experiment or will we see a resurgence in the use of encryption by spammers?
Random non-email-spam-related aside: There have been widespread reports from the Asterisk open-source PBX community that spammers are attempting to gain access to Asterisk PBX through brute-force attacks originating from hosts within the Amazon EC2 cloud computing environment. The Asterisk-users mailing list has an active discussion on the topic.
Some thoughts from the email spam perspective:
The VoIP community seems to be responding (and quickly) with blocking tools as a first line of defense against these attacks. How long will it be before spammers get the message and lower their per-IP volume to evade detection as they have done with email spam?
When will we see RBLs emerging to assist with this problem?
Amazon seems to be handling the abuse reports poorly – at least that’s the perception of the Asterisk users. When will the community get together to establish a VoIP abuse reporting framework similar to ARF?
Is this problem affecting only the open source PBX crowd? Many of the Asterisk users seem to be running Asterisk on DSL connections etc.. It’s unlikely that vendors will help them.
Security researchers at millw0rm.com recently discovered a shockingly straightforward vulnerability in the DD-WRT open source firmware that is commonly installed on Linksys routers. The vulnerability enables one-liner ownership of DD-WRT boxes. For example, typing the following into your browser while within a hot spot served by a DD-WRT router will provide root shell access on port 5555:
With the spaces removed, you can see this is just a call to the venerable netcat (i.e. “nc”) command, which sets up a tiny server on port 5555 running the shell (/bin/sh). Once the shell service is thus initiated, you can log in using telnet and execute commands – note that the shell prompt is not displayed:
Worse, it’s possible to exploit this vulnerability by simply publishing IMG tags around the web that unsuspected people will visit using their browser while sitting in a hot spot serviced by a vulnerable DD-WRT wireless router. The IMG tag’s src URL just has to provide a netcat shell server as described above, and the router is instantly vulnerable to an attacker.
No doubt spammers and bot herders will be taking advantage of this vulnerability to create armies of wireless routers that they can press into service as spam zombies. The great thing about Linux-based routers as spamming machines is that they run at the edge of the firewall, enabling easier establishment of peer-to-peer command and control networks.
Over the next few months, we are going to be writing a series of posts about filtering outbound spam as it exits (or attempts to exit) an ISP’s network. This series of posts will be tracking our actual efforts as we work with a mid-sized ISP (~1M users) to help them prevent botnet-originated spam from leaving their subscriber network by forcing port 25 traffic to pass through an intermediate filtering layer running Traffic Control.
To pique your interest in the upcoming posts, you may be interested to know that we will be implementing Traffic Control 4.x in a fully transparent configuration – intercepting all of the ISP’s outbound port 25 traffic, scanning it for spam using an efficient filter, applying some traffic shaping and blocking, and then forwarding it to the destination mail server. We’ll be talking about how we can proxy SMTP AUTH requests, and how SSL (i.e. STARTTLS) can be accommodated in such a situation without adversely affecting users.
Questions before we get started? Just leave us a comment.
POST
FIRST
PREVIOUS
