Last week, according to the BBC, South Korea’s Internet and Security Agency began encouraging ISPs to block port 25 to limit the quantity of botnet spam emanating from the country. South Korea has long had a reputation as a haven for botnet spam, most likely because of the large number of Internet users in the country, and the extremely high quality and low cost of their broadband access. The recommendation to block port 25 will probably improve things in South Korea, if the ISPs get around to implementing this change. I’m not sure how influential the regulator is in that country, but if it’s like other developed countries, the ISPs are likely to drag their feet to avoid affecting users negatively.
“Outbound spam filtering is all about ensuring reliable email delivery. If your organization counts on email delivery, then you should invest in outbound spam filtering.”
We’ve written a new white paper that discusses the need for a multi-layered approach in dealing with outbound email abuse (i.e. outbound antispam). The layers are thus:
Accurate content filtering – using a great spam filter to tag and potentially reject spam messages before they leave the network;
Local reputation management and policy enforcement – keeping track of the reputation of “senders” in your network, and preventing “bad” users from getting too abusive; and,
IP address management – moving traffic out through different blocks of IP addresses depending on the reputation of the sender.
The idea of rolling out these techniques is to do on the sending side what receivers are doing on the receiving side – except with the benefit of knowing more about your senders, such as their “account id” or “phone number” depending on what kind of network or service you’re operating. We’d love to hear your feedback on this new white paper, so go ahead and download A Multi-Layered Approach to Effective Outbound Spam Protection and let us know what you think.
I popped open Excel and generated some stats porn for everyone today.
One of the interesting things we track here at MailChannels is the positioning of the world’s worst spam sources on the world’s best blacklists. The chart above shows the number of blacklist entries on the Composite Blocking List (CBL – link) for each of the top-15 spam sending networks on the Internet. The CBL tracks botnet infections (excellent statistics are available on the CBL web site) by analyzing spam traffic aimed at its extensive honeypot network, and then lists the IP addresses from which this spam traffic originates. The listings are automated, and listings can be easily removed by ISPs through a web page once the bot problem has been resolved. Listings that are not manually removed in this manner do eventually time out on their own.
I suppose one of the interesting things about this chart is that despite the fact that spam almost disappeared over the holidays (see our previous post), the number of CBL listings produced by each of these networks stayed relatively constant during that time period (our chart starts roughly in late November 2010). I’m impressed by the apparent efforts of the folks at vnnic.net.vn (Vietnam Post and Telegraph Company) to clean up their act, resulting in a substantial drop in listings during the time period under analysis. But for most of these providers, it seems that business as usual continues to prevail when it comes to removing bot infections from their networks.
USA vs. Russia vs. Thailand vs. China
The largest spam sources don’t always come from the largest countries. For a variety of reasons, the United States (population 308,745,538) has far fewer bot infections listed in the CBL’s top-100 spamming networks list than the much smaller country of Thailand (population 65,998,436). Russia tops this comparison, however, with nearly 10-times the number of CBL listings in the top-100 spamming networks list during the time period under analysis.
The Worst Spamming Countries
In economic news, we often hear of the “BRIC”, which refers to Brazil, Russia, India, and China. The BRIC nations are fast-growing, with large, young populations, and apparently are also a great source of spam. If we look at the number of spamming networks from each country that are listed in the CBL’s top-100 spamming networks list, we find Russia on top, with India in second place, Brazil in third trailing not far behind, and .. actually, China doesn’t even make the list. China would be on the list weren’t it for the fact that Internet access in that country is highly concentrated among a small group of massive ISPs.
Again, I find it strange that Thailand makes this list, considering its very small population. Armenia is also a surprise – with a population just over 3M, you have to wonder how they manage to get so many networks into the top-100 list of spam sources.
Conclusions
It’s not news (at least, not to me) that the world’s largest spam sources are developing nations. Developing countries are often many years behind developed countries in their acquisition of technology because vendors tend to visit these countries last after developing what is perceived to be more profitable first-world markets initially. We humbly assert that MailChannels is doing its part in the developing world to reduce the spam problem (read our recent case study on outbound spam control at Cambodia’s Ezecom for reference). As we succeed in landing more outbound spam control customers in these markets, my great hope is that the CBL list of 2011 looks a lot better in all respects than it did at the tail end of 2010.
Cloud hosting services that rent "virtual servers" are one of the largest sources of spam on the Internet.
We recently implemented our transparent outbound spam protection solution with a provider of “cloud” or “virtual” hosting services. I thought I would write a blog post anonymously discussing some of the interesting results and observations we can make regarding their outbound spam problem. One day, I hope to form this into a case study, which we will publish on our web site.
The Cloud Provides a Great Platform for Spammers
We normally think of spam as originating from “botnets” of compromised personal computers. This is not a bad assumption to make, because it seems the majority of the world’s 300B spam messages do in fact still originated from compromised PCs. But botnets are not the only game in town. In fact, the emergence of cloud hosting services such as Amazon EC2, Rackspace Cloud, and others have provided a powerful, easy to use new platform for spammers to abuse.
A cloud service typically provides a given unit of CPU, hard disk, and network resources at an hourly or monthly rental rate. In practice, this means a Linux machine instance hosted within a virtual machine of some sort. The cloud service provider chops up thousands of physical machines, hard disks, etc.. essentially selling them piecemeal to customers to reap a return on their capital investment. Here’s how spammers abuse cloud services:
The blobs in Red indicate steps taken by the spammer, and the consequences of spamming. The blogs in blue indicate the hosting provider’s steps. Here’s the process in plain English:
Spammer uses a stolen credit card to rent a quantum of cloud hosting infrastructure. The owner of the stolen credit card won’t likely find out about the fraud until his or her statement arrives in 6 to 8 weeks’ time.
The spammer installs spamming software on the cloud hosting machine, and begins sending out spam.
Huge volumes of spam are delivered from the hosting machine. We have seen single machine instances that send more than a million messages a day.
The hosting provider receives spam complaints from the rest of the Internet (through feedback loop systems like AOL’s SCOMP and ARF messages from other providers).
Eventually, the credit card holder complains about the fraudulent transaction the spammer made, and requests a chargeback from the credit card company.
The chargeback costs the cloud hosting provider anywhere from $50 to $100 or more (smaller hosting providers can pay much more if the fraud rate increases). Credit card issuers can also require a “holdback” if fraud rates become particularly high; and this holdback amount can cripple the hosting provider by tying up valuable cash resources essentially on a permanent basis.
How Hosting Spam Differs from Botnet Spam
For spammers, the cloud hosting environment is in many ways superior to using a botnet, and in some ways is inferior. The superior features of a cloud hosting environment include:
Static IP addresses: Cloud hosting providers will often offer static IP addresses, which are considered more reliable by email receivers because they are more reasonably associated with legitimate email servers;
Great bandwidth: Cloud hosting providers offer awesome upstream network performance, easily beating the performance and throughput of any consumer-grade ISP network;
A flexible and easy to use operating system: A cloud hosted server provides the spammer with a complete Linux operating system, on which sophisticated spamming tools can be installed. By contrast, a compromised PC presents a very challenging software platform because the spamming software must stealthily reside within an operating system that is designed for real-time use by an interactive user; and,
24×7 around the clock operation: Unlike compromised PCs, which are often turned off at night, cloud hosting services remain up 24 hours a day, 7 days a week.
As a result, we can observe some marked differences between spam that is delivered from cloud hosting networks versus spam that is delivered via a botnet. To illustrate, let’s look at a couple of graphs to show how hosting networks are “24×7″ whereas ISP networks operate mostly in the daytime – essentially giving a 16-hour advantage to the hosting networks as a spamming platform:
First, here is a graph showing the sidereal (i.e. daytime-only) nature of SMTP traffic from an ISP’s subscriber network:
A week's worth of SMTP traffic from an ISP subscriber network shows how the traffic surges during daylight hours and fades at night and on the weekend. This literally corresponds to people having their machines turned on in the daytime, and then turning them off at night.
This data is from a customer in Asia, so the time zone shifts things a bit to the right of where they would be if they were in our time zone here in Vancouver, Canada.. but you get the idea. People don’t generally have their computers turned on at night, and as a result, the spammers don’t have access to them to send email. In fact, this customer is located in a developing nation in Asia, where the daytime/night-time difference is even more dramatic than with ISPs in developed countries (because there literally isn’t power at night).
Now, let’s look at a graph from a cloud hosting network:
A week's worth of SMTP traffic from a cloud hosting provider shows the 24x7 nature of hosting networks versus ISP networks.
The hosting provider’s network sends email 24×7. The spikes in traffic show times when a single machine leveraged the enormous capacity of a cloud hosting instance to establish upwards of 8,000 SMTP connections per second.
Conclusions and Recommendations
Spammers threaten the viability of cloud hosting infrastructure by abusing these services and generating excessive customer support and fraud-mitigation costs. Having now seen a good sample of traffic from cloud hosting networks and comparing that against traffic from ISP networks, I think we can make the following recommendations to cloud hosting providers:
It’s useful to have a real-time transparent spam filtering solution in place – waiting for feedback from the Internet about the spammers within a cloud hosting network introduces a delay that is exploited by spammers to great effect. If you can catch the spam in real-time — even if your filtering isn’t perfect — you’ll make your network less attractive to spammers.
Reporting is really important – even with a transparent outbound spam filtering solution, you will still need to know which customers are the fraudsters. This is harder than it sounds, and I will try to cover this topic in a future blog post.
Until then, I hope you enjoyed reading this post and look forward to your comments and questions.
On September 30, 2010, MailChannels CEO Ken Simpson talked about conventional anti-spam techniques are ineffective at stopping outbound spam. In video 3 of a 5-part series, he answered:
How are service providers trying to stop outbound spam?
- Using inbound anti-spam techniques for scanning outgoing email
- Blocking port-25
- Blocking entire IP ranges
- Manual handling by support staff
On September 30, 2010, MailChannels CEO Ken Simpson talked about how email spam gets sent out onto the Internet. In video 2 of a 5-part series, he answered:
Where does spam spam come from?
The short answer, botnets through three key channels:
- directly out through email servers via port-25
- sending out through the ISP’s mail relay
- compromised webmail
Over the coming weeks, we’ll be uploading a series of video posts by MailChannels CEO Ken Simpson on the topic of outbound spam. These informal videos are meant to explain the outbound spam problem, and its solutions, in a plain and easy to understand manner. We would appreciate your comments, and suggestions for new videos we might produce.
We recently took a look at about 9 million delivery attempts through one of our outbound spam filtering customers’ systems and compiled a list of the most common spam connection rejection messages. I’ll share the list with you later in this blog post. But first, a bit of technical background for the uninitiated.
Most mail servers are configured to reject connections from IP addresses that appear on a black list. A notable exception to this would be Google, who through the sheer might of their infrastructure appear to accept connections from anyone, perhaps in order to beef up their spam folder size (to make it appear like they are doing more than they really are to reduce spam). In any case, if your SMTP connection is rejected because your IP address is found on a blacklist, you will get an error message from the receiving mail server that looks like the following:
Mail from 192.0.32.10 not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/errors/550-bl23.html [550]
This rejection message is packaged up into a non-delivery receipt by your mail server, which sends this to your mailbox. You then hopefully click on the link, read Yahoo’s excellent error message documentation, and then take steps to have your IP address removed from the Spamhaus XBL as instructed.
I wanted to see which messages were the most popular, so I used our powerful log indexing system to pull out an hour’s worth of rejection notices. I then ran these through a very simple Map-Reduce script to count up the error responses, stripping out random variations like IP addresses and transaction IDs. What resulted is the following list of the world’s most popular blacklists — at least, from the perspective of error responses:
25% 5.7.1 [BL21] Connections will not be accepted from 1.2.3.4
10% Mail from 1.2.3.4 not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/errors/550-bl23.html [550]
8% You are not allowed to connect.
8%
5% Service unavailable; Client host [1.2.3.4] blocked using Trend Micro RBL+.Please see http://www.mail-abuse.com/cgi-bin/lookup?ip_address=1.2.3.4
3% Mail from 1.2.3.4 not allowed - 5.7.1 [BL21] Connections not accepted from IP addresses on Spamhaus PBL; see http://postmaster.yahoo.com/errors/550-bl21.html [550]
3% IP:1.2.3.4 - A problem occurred. (Ask your postmaster for help or to contact tosa@rx.t-online.de to clarify.)
2% Transaction failed. For explanation visit http://freemail.web.de/reject/?ip=1.2.3.4
2% No SMTPd here
1% 5.7.1 5.7.1 Client host rejected: Dynamic IP addresses are blocked. Please contact your email provider.
1% Denied by policy
1% Email from 1.2.3.4 is currently blocked by Verizon Online's anti-spam system. The email sender or Email Service Provider may visit http://www.verizon.net/whitelist and request removal of the block. 100604
1% RBL rejection: http://www.spamhaus.org/query/bl?ip=1.2.3.4
1% 5.5.0 Improper use of SMTP command pipelining
1% Mail from 1.2.3.4 not allowed - VS98-IP1 deferred - see http://help.yahoo.co.jp/help/jp/mail/anti-spam/anti-spam-24.html
1% 5.7.1 CCRX 1.2.3.4: Connection refused. Your IP address is blocked(anti-spam). If you need [truncated]
1% 5.7.1 service refused. Client host 1.2.3.4 blocked for spamming issues. Adresse IP source 1.2.3.4 bloquee pour incident de spam. Ref http://r.orange.fr/r/Oassistance_adresserejetee .
1% Blocked - see https://support.proofpoint.com/dnsbl-lookup.cgi?ip=1.2.3.4
1% 5.7.1 service refused. Client host 1.2.3.4 blocked for spamming issues. More information available at http://help.orange.c [truncated]
1% 5.7.1 Service unavailable; Client host [1.2.3.4] blocked using Trend Micro RBL+.
I’m amazed at how dominant Yahoo is in this list. Perhaps we chose a window of time during which this particular ISP network was hammering them especially hard in preference to the other networks.
POST
FIRST
PREVIOUS
