We recently took a look at about 9 million delivery attempts through one of our outbound spam filtering customers’ systems and compiled a list of the most common spam connection rejection messages. I’ll share the list with you later in this blog post. But first, a bit of technical background for the uninitiated.
Most mail servers are configured to reject connections from IP addresses that appear on a black list. A notable exception to this would be Google, who through the sheer might of their infrastructure appear to accept connections from anyone, perhaps in order to beef up their spam folder size (to make it appear like they are doing more than they really are to reduce spam). In any case, if your SMTP connection is rejected because your IP address is found on a blacklist, you will get an error message from the receiving mail server that looks like the following:
Mail from 192.0.32.10 not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/errors/550-bl23.html [550]
This rejection message is packaged up into a non-delivery receipt by your mail server, which sends this to your mailbox. You then hopefully click on the link, read Yahoo’s excellent error message documentation, and then take steps to have your IP address removed from the Spamhaus XBL as instructed.
I wanted to see which messages were the most popular, so I used our powerful log indexing system to pull out an hour’s worth of rejection notices. I then ran these through a very simple Map-Reduce script to count up the error responses, stripping out random variations like IP addresses and transaction IDs. What resulted is the following list of the world’s most popular blacklists — at least, from the perspective of error responses:
25% 5.7.1 [BL21] Connections will not be accepted from 1.2.3.4
10% Mail from 1.2.3.4 not allowed - 5.7.1 [BL23] Connections not accepted from IP addresses on Spamhaus XBL; see http://postmaster.yahoo.com/errors/550-bl23.html [550]
8% You are not allowed to connect.
8%
5% Service unavailable; Client host [1.2.3.4] blocked using Trend Micro RBL+.Please see http://www.mail-abuse.com/cgi-bin/lookup?ip_address=1.2.3.4
3% Mail from 1.2.3.4 not allowed - 5.7.1 [BL21] Connections not accepted from IP addresses on Spamhaus PBL; see http://postmaster.yahoo.com/errors/550-bl21.html [550]
3% IP:1.2.3.4 - A problem occurred. (Ask your postmaster for help or to contact tosa@rx.t-online.de to clarify.)
2% Transaction failed. For explanation visit http://freemail.web.de/reject/?ip=1.2.3.4
2% No SMTPd here
1% 5.7.1 5.7.1 Client host rejected: Dynamic IP addresses are blocked. Please contact your email provider.
1% Denied by policy
1% Email from 1.2.3.4 is currently blocked by Verizon Online's anti-spam system. The email sender or Email Service Provider may visit http://www.verizon.net/whitelist and request removal of the block. 100604
1% RBL rejection: http://www.spamhaus.org/query/bl?ip=1.2.3.4
1% 5.5.0 Improper use of SMTP command pipelining
1% Mail from 1.2.3.4 not allowed - VS98-IP1 deferred - see http://help.yahoo.co.jp/help/jp/mail/anti-spam/anti-spam-24.html
1% 5.7.1 CCRX 1.2.3.4: Connection refused. Your IP address is blocked(anti-spam). If you need [truncated]
1% 5.7.1 service refused. Client host 1.2.3.4 blocked for spamming issues. Adresse IP source 1.2.3.4 bloquee pour incident de spam. Ref http://r.orange.fr/r/Oassistance_adresserejetee .
1% Blocked - see https://support.proofpoint.com/dnsbl-lookup.cgi?ip=1.2.3.4
1% 5.7.1 service refused. Client host 1.2.3.4 blocked for spamming issues. More information available at http://help.orange.c [truncated]
1% 5.7.1 Service unavailable; Client host [1.2.3.4] blocked using Trend Micro RBL+.
I’m amazed at how dominant Yahoo is in this list. Perhaps we chose a window of time during which this particular ISP network was hammering them especially hard in preference to the other networks.
Tags: blacklists, google, outbound, outbound spam, spam, spamhaus, yahoo
June 12th, 2008
Posted in Uncategorized
With constantly growing spam volumes, large email providers have been forced to take measures to reduce spam’s impact on their infrastructure and on their customers. Unprotected email systems are easily crippled by spam outbreaks and it doesn’t make sense to overbuild capacity to meet what-if situations.
One way large email providers protect their systems is through the use of a reputation database along with mail architectures which use the database to rate-limit or block emails. AOL, Yahoo!, and others maintain proprietary reputation systems. The following best practices can help maintain your good reputation – improving your mail deliverability not just to Yahoo! and AOL but to everywhere.
If you are having problems with your emails bouncing or being treated as junk, there are three steps you can take which can improve your deliverability:
A) Check the reverse DNS entry for your outbound mail server’s IP address
- Make sure the forward and reverse match.If your IP resolves to example.com, make sure that one of the addresses that example.com resolves to is your IP.
- Make sure its set for your server specifically, preferably tothe domain you are sending mail from.
- Don’t use an address that looks dynamic. A reverse DNS of d192-168-0-1.example.com is considered a much likelier source of spam then “mail.example.com”.
B) Avoid having your server look like a source of spam
- Old email accounts are often forwarded to a new address. If the address your server forwards to is eventually de-activated, does your email server stop forwarding messages to it? Many won’t automatically. This will mean your server is regularly sending email to an invalid recipient – something that spammers do.
- If possible, forward email after spam filtering. This can help reduce the impression that it’s your server that originated the spam. It may also help keep your mail queue from filling up with messages which aren’t accepted by the server you are forwarding to.
- Ensure good list hygiene practices. Whether its someone in marketing sending email from their desktop or a properly managed list using list management software, your responsibilities are the same. Only email people who actually want to hear what it is you are saying to them. Promptly handle and always respect any requests to unsubscribe. Unsubscribe addresses which bounce repeatedly. Comply with CAN-SPAM.
As an email server administrator, configuring your server correctly can go a long way to following these guidelines, but that is only one piece of the puzzle. Your users needs to be educated, and your mail server’s logs need to be reviewed periodically. Your pro-active approach will help avoid having your user’s productivity impacted when they can’t email their contacts.
Tags: CAN-SPAM, email, smtp, yahoo
April 7th, 2008
Posted in Uncategorized
Once Promising Proposals for a Final Ultimate Solution to the Spam Problem (FUSSP)
“Two years from now, spam will be solved.”
That was Bill Gates’ famous pronouncement back in 2004. Microsoft, Yahoo and the open source community devised two techniques that they believed would eradicate spam. The first was sender authentication, which allowed email senders to provide a list of the servers permitted to send email for users within their domain. The idea was that sender authentication would eliminate spammers spoofing legitimate email addresses, and allow for the creation of a permanent, ironclad white list of trustworthy domains that never send spam, thus allowing recipients to simply block everything not on the white list and end spam forever.
Another idea pitched in 2004 was the computational challenge. Senders would, upon connecting to a receiving email server, have to spend considerable CPU cycles computing the answer to a mathematical challenge provided by the receiving server. Bill Gates believed this approach would stop spam by making it cost too much to send the high volumes of email required to make spamming profitable.
Unfortunately, neither sender authentication nor the computational challenge technique resolved the spam problem. Computational challenges were rejected as being too costly for legitimate bulk email senders (airlines, banks, open source mailing lists, etc.) And sender authentication while eventually enjoying wide-spread adoption in the form of DKIM and SenderID, proved prone to errors. As as result it has remained useful mostly for the acceptance of legitimate email and phishing protection rather than the rejection of spam.
By 2005, what the anti-spam community was getting right was content filtering. When spam filters had reached above the 90 per cent accuracy level, spam transitioned from a problem of content to a problem of volume, the spammers simply send more spam. And they can do this because the recipient pays the cost of content filtering rather than the spammer.
The cost of a resource-consuming filtering system increases during high traffic loads. If you block spam content, spammers will find new ways to get around it. Bill Gates was right, the only way to stop them is to create difficulty by making spam too costly to send. If you do spammers are left to find new targets that are easier to hit.
NEXT: Post #4 Spamonomics: The Economics of Spamming
PREVIOUS: Post #2 Prohibition Induces “Botlegging”
Tags: accuracy, anti-spam, bill gates, content-filters, dkim, economics, high traffic loads, microsoft, spam, spammers, spamonomics, yahoo
