Reducing Spam to Nearly Zero with PTR Record Filtering

Blocking spam is an arms race between spam detection and detection avoidance techniques. Lately spammers had the upper hand but the tide has turned with new PTR record blocking techniques. This is how implementing PTR record filtering has reduced our spam to nearly zero.

There are two general methods of detecting and blocking spam, by IP address blacklists and by content analysis. Unfortunately spammers have long learned how to exploit the weaknesses of these methods. Exploiting content analysis was fairly easy. Ever receive image based spam or wonder why they contained random sentences? These are techniques are very effective in avoiding detection and poisoning spam keyword detection databases.

Circumventing IP blacklists was a tougher challenge. Spammers found the perfect way to stay ahead of the blacklists, Windows’ weak security and the growth of broadband adoption in the home. Millions of unprotected computers connected to the Internet were waiting to be exploited as spam machines. Automated exploit propagation allowed spammers to add zombies faster than system admins could blacklist them. Combined with excellent content filtering circumvention spammers had the spam equivalent of a nuclear bomb.

PTR record filtering, a new technique, has proven to be an effective defense against the horde of spam zombies. A PTR record is a reverse DNS entry, resolving an IP address to a host name. PTR filtering works by blocking messages coming from host names that match a specific pattern. This has a huge advantage over traditional blacklists because a single pattern can match thousands of IP addresses. In contrast a blacklist requires a person to add each individual IP address separately.

Let’s take a look at a real world example. I recently added PTR record filtering to Sutton’s spam firewalls. This simple regular expression “X-Barracuda-Connect: s\d+.*.*shawcable.net...” matches all of Shaw’s home cable connections. In 24 hours it blocked 730 spam messages that would have slipped past the traditional filters. By expanding and including patterns for other cable and DSL networks more than 19,000 false negatives were blocked.

Let me put that in perspective:

1. We average about 400K to 500K emails entering our system daily (spam, ham, viruses, etc).
2. Before PTR record filtering approximately 40K to 50K was allowed through daily.
3. After, the number of allowed messages dropped between 20K and 30K a day.
4. That is approximately a 40% to 50% reduction. That is huge.

As amazing the results are, they are still far from perfect. The current Barracuda firmware (3.4.10.087) has problems correctly resolving PTR records every so often. This is a major bug in their software and when fixed it will only make PTR based blocking that much more effective. Unfortunately this bug also affects the good servers so I can’t even tag messages with [BULK] for user level filtering.

With such a huge drop in allowed messages false positives are also a big concern. In my testing I haven’t seen any. There is a risk of blocking people legitimately using their consumer level broadband to send email. In my data I found this to be very rare. The amount of spam coming from zombie machines is so high that the trade off is still worth it. People who are savvy enough to run their own mail relay at home should hopefully be savvy enough to figure out why their emails are getting blocked, especially as PTR record blocking becomes more popular.

I’ve uploaded two CSV files containing the data exported from our firewall. The first contains info on all 19,000+ spams blocked by PTR filtering in the past 24 hours and the second contains the 730 blocked from Shaw. These should be useful if you’re curious about running your own analysis on the data.

[tags]spam, ptr, barracuda, email, dns, dns blacklists, image spam[/tags]

Tagged as: Linux