outbound spam
How do Cloud providers deal with outbound spam?
When Amazon Web Services (AWS) used a live webcast to announce their new DynamoDB system, with Urz Wendler looking just shy of Steve Jobs (although a few pounds heavier), I knew that the cloud had finally arrived. One of the new buzz words surrounding the cloud (there are many) is IaaS, which stands for “Infrastructure as a Service”. Amazon Web Services was perhaps the first IaaS provider. Among them, VPS.net and RackSpace were early entrants. But as the market matures, an increasing number of companies now provide an IaaS, including HP, who relatively recently launched the HP Cloud. Why so many providers? Shouldn’t there be consolidation in the IaaS space as it matures?
No, there shouldn’t be consolidation yet, and this is all thanks to the widespread availability of excellent, open source IaaS offerings like Cloud.com and OpenStack, as well as commercial offerings from companies like VMWare and even startups like OnApp, whose own system powers the massive VPS.net IaaS. These IaaS packages make it easy for someone to set up their own IaaS, in much the same way that Zimbra made it easy for anyone to start offering a great hosted email service.
Of course, as more public IaaS clouds sprout up, we’re seeing more inquiries from IaaS providers who are looking for a way to combat spam and other forms of abuse originating within their newly minted IaaS clouds. Because it’s so easy to set up new virtual servers and other resources with a cloud environment, public IaaS cloud operators are besieged by bad guys, who use fraudulent means to open new accounts, set up spamming boxes (and other bad things like click fraud, child porn hosting, etc.) and begin blasting away at the good reputation of the IaaS provider.
As we see these IaaS services mature, I’m confident we will also see increased demand for services and products to combat fraud within the cloud. And, of course, I look forward to helping these companies with that problem.
Tags: cloud, iaas
Last week, according to the BBC, South Korea’s Internet and Security Agency began encouraging ISPs to block port 25 to limit the quantity of botnet spam emanating from the country. South Korea has long had a reputation as a haven for botnet spam, most likely because of the large number of Internet users in the country, and the extremely high quality and low cost of their broadband access. The recommendation to block port 25 will probably improve things in South Korea, if the ISPs get around to implementing this change. I’m not sure how influential the regulator is in that country, but if it’s like other developed countries, the ISPs are likely to drag their feet to avoid affecting users negatively.
Tags: block, korea, outbound spam
After a very quiet summer, it seems that the spam bots have awakened. We recently noticed a spike in blacklistings; after checking the usual data sources, it seems the spike is widespread and indicative of one or more spambot networks getting back into action. The graph above is from the CBL (source: cbl.abuseat.org), which is one of the world’s better botnet blacklists.
Is this another “back to school special” as we have seen in previous years, where the spammers go to sleep over the summer, only to turn their machines back on in time for mom and dad to come home from vacation (and start buying pills)?
Still, despite the recent uptick, botnet spam volume is still well below the peak in early 2010, which was as much as 6 times higher than present volumes.
I have a question for the spamosphere: Why is the tiny UCEPROTECT blacklist so influential with large telecommunications providers? Nearly every provider I speak to is enormously concerned about their reputation on the UCEPROTECT network. This concern seems strange to me, considering the small penetration of UCEPROTECT versus other blacklists like Spamhaus, SORBS, and Trend Micro MAPS+.
Reviewing millions of delivery attempts at a large service provider, I found literally zero references to UCEPROTECT in SMPT error messages. Here is a summary of the most popular blacklist service URLs, found by searching the outbound SMTP filtering logs of an ISP and a major cloud hosting provider:
As you can see – there’s no UCEPROTECT to be found. In fact, I grepped the logs of millions of SMTP connection attempts from many different networks, and could not find examples of receivers indicating that UCEPROTECT was the reason that a connection or message had been blocked.
So, then, my question to you all is: Why is UCEPROTECT taken so seriously? Because, they definitely are.
In March, I had the pleasure of visiting the great cities of Hong Kong, Manila and Singapore. Since I am on a mission to end the global spam problem by helping ISPs plug up botnet spam through transparent SMTP filtering, and seeing as the CBL ranks Asia as one of the worst emitters of spam on the planet, I figured I had better pay this region a visit. My trip was thoroughly interesting and enjoyable. Whether it was battling Manila’s rush hour traffic and smog, cheering and boozing with a group of bankers at the Rugby 7′s in Hong Kong, or admiring Singapore’s breathtaking new architecture, I learned a few key things about Asia: it’s the future, it’s wireless, and there are lots of poor people. And this is perhaps why so much spam comes from Asia.
Hong Kong: Mobile World
My Asian home base during March was in Hong Kong, where I met with the major mobile carriers to discuss outbound spam. Non-work highlights included dancing at the various night clubs of Lan Kwai Fong, shopping in Mong Kok’s vibrant street market, and swimming with the masters club at Wan Chai Training Pool. Not to mention thoroughly enjoying the Rugby 7′s (Google it if you are so inclined).
Hong Kong has what I would consider to be the world’s most competitive mobile telephone market. In Hong Kong, seven major carriers compete for customers in this the world’s most “vertically oriented” city. Mobile broadband is ubiquitous, and wireless service is incredibly cheap. In Hong Kong, you can buy pre-paid (i.e. no commitment) wireless service for about USD $3/day, which includes unlimited data, unlimited voice, and unlimited texting. As a visitor, this is just mind-blowing. And did I mention the pre-paid SIM card, which costs about USD $15, includes $15 worth of credit? Hong Kong wireless service is so competitive that most people have more than one mobile phone – penetration of wireless is at 170% of the population (source).
Trouble is, Hong Kong’s networks originate a great deal of spam (data courtesy of SenderBase). The pie chart at right summarizes the IP address reputation of all of the email-sending IP addresses owned by a major Hong Kong-based mobile operator. The pie chart represents over 5,000 IP addresses, which is a small slice of the IP address space owned by this operator about which SenderBase is aware.
Fortunately, Hong Kong operators are aware of these issues and are taking steps to contain outbound spam through a variety of techniques, including transparent spam filtering and outbound spam filtering at mail relays. This is much more, sadly, than one can say for other operators in the region, which originates a very large proportion of the world’s spam.
I hope to return to Hong Kong in the very near future and am looking forward to exploring Lan Kwai Fong once more, as well as making greater use of the fantastic and cheap Public Light Bus service (people from Hong Kong will laugh at this).
At the RSA Conference 2011, Microsoft’s Scott Charney gave a talk proposing that ISPs should quarantine compromised customer accounts that are spewing spam.
Charney argues that this can be done with existing technology using a system that checks a computer’s “health” before granting unfettered access to the Internet. In general, we think this approach is a good idea, but as with many “big picture” security ideas it has a few flaws. The main flaw is, how do you know that a system is clean? Perfectly clean systems can be infected with zero-day malware and begin spewing out spam and malware on a moment’s notice. What value would the health certificate have in this case? Another flaw is that the security certificate system would have to be “signed” by someone. Who would create certificates, and how would the Internet community know that they can be trusted?
Fortunately, because compromised computers tend to be used for spamming and other obvious network-borne attacks, there is an easier and immediately implementable solution that doesn’t require certificates to provide great protection to the Internet community. The solution involves monitoring the external behaviour of machines through network sensors and filters, and then clamping down on a machine’s access to Internet resources (i.e. bandwidth and ports).
MailChannels specializes in outbound spam filtering, so we can comment on detecting spam in the network. But other companies offer solutions for detecting other kinds of nefarious activity – particularly the accessing of botnet command and control systems.
If you’re not familiar with what we do, consider this a quick refresher. Or, if you’re interested in understanding more about how to protect the reputation of networks from compromised customer accounts sending spam, please read on.
Outbound Filtering
According to reputation security networks, most ISPs in the world have a chunk of IP addresses that are bad – some even as high as 99%. When you take a closer look at “Poor” addresses, we find many of them are listed on blacklists. Anyone sending email from within these IP ranges will be blocked by most of the Internet.
Outbound spam filtering allows ISPs to take immediate action within seconds, and completely automate the process of improving your reputation before botnet infections get you blacklisted.
Transparency
Our outbound spam filtering operates transparently so you can deploy without major configuration changes. It transparently intercepts all port-25 traffic coming out of the network, and passes the traffic through one or multiple content filters from leading vendors. SMTP AUTH and SSL encryption is fully supported so the privacy of conversations are respected.
–
How are you identifying fraudulent customer accounts?
I popped open Excel and generated some stats porn for everyone today.
One of the interesting things we track here at MailChannels is the positioning of the world’s worst spam sources on the world’s best blacklists. The chart above shows the number of blacklist entries on the Composite Blocking List (CBL – link) for each of the top-15 spam sending networks on the Internet. The CBL tracks botnet infections (excellent statistics are available on the CBL web site) by analyzing spam traffic aimed at its extensive honeypot network, and then lists the IP addresses from which this spam traffic originates. The listings are automated, and listings can be easily removed by ISPs through a web page once the bot problem has been resolved. Listings that are not manually removed in this manner do eventually time out on their own.
I suppose one of the interesting things about this chart is that despite the fact that spam almost disappeared over the holidays (see our previous post), the number of CBL listings produced by each of these networks stayed relatively constant during that time period (our chart starts roughly in late November 2010). I’m impressed by the apparent efforts of the folks at vnnic.net.vn (Vietnam Post and Telegraph Company) to clean up their act, resulting in a substantial drop in listings during the time period under analysis. But for most of these providers, it seems that business as usual continues to prevail when it comes to removing bot infections from their networks.
USA vs. Russia vs. Thailand vs. China
The largest spam sources don’t always come from the largest countries. For a variety of reasons, the United States (population 308,745,538) has far fewer bot infections listed in the CBL’s top-100 spamming networks list than the much smaller country of Thailand (population 65,998,436). Russia tops this comparison, however, with nearly 10-times the number of CBL listings in the top-100 spamming networks list during the time period under analysis.
The Worst Spamming Countries
In economic news, we often hear of the “BRIC”, which refers to Brazil, Russia, India, and China. The BRIC nations are fast-growing, with large, young populations, and apparently are also a great source of spam. If we look at the number of spamming networks from each country that are listed in the CBL’s top-100 spamming networks list, we find Russia on top, with India in second place, Brazil in third trailing not far behind, and .. actually, China doesn’t even make the list. China would be on the list weren’t it for the fact that Internet access in that country is highly concentrated among a small group of massive ISPs.
Again, I find it strange that Thailand makes this list, considering its very small population. Armenia is also a surprise – with a population just over 3M, you have to wonder how they manage to get so many networks into the top-100 list of spam sources.
Conclusions
It’s not news (at least, not to me) that the world’s largest spam sources are developing nations. Developing countries are often many years behind developed countries in their acquisition of technology because vendors tend to visit these countries last after developing what is perceived to be more profitable first-world markets initially. We humbly assert that MailChannels is doing its part in the developing world to reduce the spam problem (read our recent case study on outbound spam control at Cambodia’s Ezecom for reference). As we succeed in landing more outbound spam control customers in these markets, my great hope is that the CBL list of 2011 looks a lot better in all respects than it did at the tail end of 2010.
Tags: cbl, ezecom, outbound spam, spam, statistics
A graph taken from UCEProtect's web site (www.uceprotect.net) showing a drop-off in UCEProtect listings within the ASN of a customer of ours. We did manage to create this drop-off by using our outbound filtering reputation system and carefully managing their outbound traffic; but at first the graph did not fall off like this - instead, it grew. Mysteriously.
We recently assisted a customer in Asia with a serious problem: Their entire IP range had become blacklisted by the popular UCEProtect blacklist – in the dreaded so-called “UCEProtect Level 3″ blacklist. The customer is an ISP with a large base of consumer and small business subscribers, based in a developing Asian nation. They are one of the world’s largest originators of spam, and we hope to release a case study soon that will provide more detail into what we were able to do for them with our transparent outbound spam filtering solution. But, unlike many ISPs, they are also really trying hard to do the right thing, and have invested many person-years of per-capita GDP in purchasing our transparent outbound spam filtering system.
When we deployed this customer, however, we found out – to our absolute dismay and horror – that the number of IPs in their network that were blacklisted was actually increasing rather than decreasing. This was going on despite the fact that we were filtering out 99.9% or more of the spam messages emanating from their network. To illustrate, here is a one-day graph showing the percentage of SMTP connections coming from this network that actually included message content (i.e “Scanned” – these connections had content that we could scan with the spam filter), versys the percentage of connections that contained messages that we actually delivered. As you can see, only a TINY proportion of the messages that could have been delivered were actually delivered. Most were rejected.
Only a small fraction of SMTP connections out of this customer's network actually result in message deliveries. The yellow line represents the percentage of connections actually delivered. The blue line represents the percentage of connections that contained message content.
Despite the fact that we were rejecting most of this customer’s spam traffic, they were still getting blacklisted. Why would this be happening? Shouldn’t spam traps actually receive a message before they judge that someone is sending spam? If you don’t send a spam message, but do connect to a spam trap, does this make you a spammer?
The diagram below tries to illustrate visually what I mean:
It's easy to get onto a blacklist. Just connect to a spam trap server, and optionally validate a recipient address that is unfortunately a spam trap. No DATA required.
Even if an ISP deploys a really great transparent spam filter that removes 100% of spam message content by intercepting DATA and responding with a “550 Rejected” message, the mere act of connecting to a spam trap server (1) and attempting to validate recipients (2, if one of those recipients happens to be a spam trap email address) may cause the IP to be added to the black list (3).
Is this the right approach to running a blacklist? I don’t know. In one sense, only spammers ought to be connecting to spam traps, so the mere fact that an IP connects to a spam trap server or tries to validate a spam trap address ought to cause that IP to be listed. But what if there is a filter that is diligently removing the spam? A network that removes the spam is indeed fixing the spam problem – shouldn’t blacklists reflect this in their data? What is an ISP to do if even rejecting spam traffic is ineffective in having their IPs removed from the blacklists?
The solution is simple: Make sure the blacklists don’t see traffic from the bad IPs in your network. Doing this without affecting legitimate email traffic is tricky, but possible. I hope to fill you in on how we have tackled this problem quite successfully in a future post.
Cloud hosting services that rent "virtual servers" are one of the largest sources of spam on the Internet.
We recently implemented our transparent outbound spam protection solution with a provider of “cloud” or “virtual” hosting services. I thought I would write a blog post anonymously discussing some of the interesting results and observations we can make regarding their outbound spam problem. One day, I hope to form this into a case study, which we will publish on our web site.
The Cloud Provides a Great Platform for Spammers
We normally think of spam as originating from “botnets” of compromised personal computers. This is not a bad assumption to make, because it seems the majority of the world’s 300B spam messages do in fact still originated from compromised PCs. But botnets are not the only game in town. In fact, the emergence of cloud hosting services such as Amazon EC2, Rackspace Cloud, and others have provided a powerful, easy to use new platform for spammers to abuse.
A cloud service typically provides a given unit of CPU, hard disk, and network resources at an hourly or monthly rental rate. In practice, this means a Linux machine instance hosted within a virtual machine of some sort. The cloud service provider chops up thousands of physical machines, hard disks, etc.. essentially selling them piecemeal to customers to reap a return on their capital investment. Here’s how spammers abuse cloud services:
The blobs in Red indicate steps taken by the spammer, and the consequences of spamming. The blogs in blue indicate the hosting provider’s steps. Here’s the process in plain English:
- Spammer uses a stolen credit card to rent a quantum of cloud hosting infrastructure. The owner of the stolen credit card won’t likely find out about the fraud until his or her statement arrives in 6 to 8 weeks’ time.
- The spammer installs spamming software on the cloud hosting machine, and begins sending out spam.
- Huge volumes of spam are delivered from the hosting machine. We have seen single machine instances that send more than a million messages a day.
- The hosting provider receives spam complaints from the rest of the Internet (through feedback loop systems like AOL’s SCOMP and ARF messages from other providers).
- Eventually, the credit card holder complains about the fraudulent transaction the spammer made, and requests a chargeback from the credit card company.
- The chargeback costs the cloud hosting provider anywhere from $50 to $100 or more (smaller hosting providers can pay much more if the fraud rate increases). Credit card issuers can also require a “holdback” if fraud rates become particularly high; and this holdback amount can cripple the hosting provider by tying up valuable cash resources essentially on a permanent basis.
How Hosting Spam Differs from Botnet Spam
For spammers, the cloud hosting environment is in many ways superior to using a botnet, and in some ways is inferior. The superior features of a cloud hosting environment include:
- Static IP addresses: Cloud hosting providers will often offer static IP addresses, which are considered more reliable by email receivers because they are more reasonably associated with legitimate email servers;
- Great bandwidth: Cloud hosting providers offer awesome upstream network performance, easily beating the performance and throughput of any consumer-grade ISP network;
- A flexible and easy to use operating system: A cloud hosted server provides the spammer with a complete Linux operating system, on which sophisticated spamming tools can be installed. By contrast, a compromised PC presents a very challenging software platform because the spamming software must stealthily reside within an operating system that is designed for real-time use by an interactive user; and,
- 24×7 around the clock operation: Unlike compromised PCs, which are often turned off at night, cloud hosting services remain up 24 hours a day, 7 days a week.
As a result, we can observe some marked differences between spam that is delivered from cloud hosting networks versus spam that is delivered via a botnet. To illustrate, let’s look at a couple of graphs to show how hosting networks are “24×7″ whereas ISP networks operate mostly in the daytime – essentially giving a 16-hour advantage to the hosting networks as a spamming platform:
First, here is a graph showing the sidereal (i.e. daytime-only) nature of SMTP traffic from an ISP’s subscriber network:
A week's worth of SMTP traffic from an ISP subscriber network shows how the traffic surges during daylight hours and fades at night and on the weekend. This literally corresponds to people having their machines turned on in the daytime, and then turning them off at night.
This data is from a customer in Asia, so the time zone shifts things a bit to the right of where they would be if they were in our time zone here in Vancouver, Canada.. but you get the idea. People don’t generally have their computers turned on at night, and as a result, the spammers don’t have access to them to send email. In fact, this customer is located in a developing nation in Asia, where the daytime/night-time difference is even more dramatic than with ISPs in developed countries (because there literally isn’t power at night).
Now, let’s look at a graph from a cloud hosting network:
A week's worth of SMTP traffic from a cloud hosting provider shows the 24x7 nature of hosting networks versus ISP networks.
The hosting provider’s network sends email 24×7. The spikes in traffic show times when a single machine leveraged the enormous capacity of a cloud hosting instance to establish upwards of 8,000 SMTP connections per second.
Conclusions and Recommendations
Spammers threaten the viability of cloud hosting infrastructure by abusing these services and generating excessive customer support and fraud-mitigation costs. Having now seen a good sample of traffic from cloud hosting networks and comparing that against traffic from ISP networks, I think we can make the following recommendations to cloud hosting providers:
- It’s useful to have a real-time transparent spam filtering solution in place – waiting for feedback from the Internet about the spammers within a cloud hosting network introduces a delay that is exploited by spammers to great effect. If you can catch the spam in real-time — even if your filtering isn’t perfect — you’ll make your network less attractive to spammers.
- Reporting is really important – even with a transparent outbound spam filtering solution, you will still need to know which customers are the fraudsters. This is harder than it sounds, and I will try to cover this topic in a future blog post.
Until then, I hope you enjoyed reading this post and look forward to your comments and questions.
Regards,
Ken
Tags: hosting, outbound, outbound spam, outbound spam protection, spam, transparent filtering
POST
FIRST
PREVIOUS
