Are we losing the war on spam? Is the war on spam a war we can win? Is there any reason for hope?
When I learned how much spam was hitting our servers at O’Reilly, I decided to ask several long-time Internet luminaries these questions. Was the situation as bad as I thought it might be? In short, the answer is yes, which only makes me wonder why more people aren’t talking about it.
Let’s first try to quantify the problem using O’Reilly’s servers as an example. I’d like to see how we compare to other organizations.
All of our incoming email goes through one of two gateways, which route mail to servers that decide to accept or reject the message. This is, of course, before the message is delivered to an O’Reilly user, who may apply additional spam filters in their email program. The bottom line, according to Bob Amen, Director of Systems Engineering at O’Reilly, nearly 95% of ALL incoming messages are spam.
Here are the amazing stats that Bob shared with me. These numbers represent a one-week snapshot (last Monday 2/26 to Sunday 3/4.)
- 829,890 SMTP connections made to our two gateway mail servers
- 904060 attempted message deliveries
- 49194 messages accepted (I think this is actually a little high due to a configuration problem with our Zimbra server.)
- 94.6% of all messages were rejected
Here’s the breakdown on how we reject messages:
- 282414 connections rejected due to bad SMTP HELO syntax
- 224722 connections rejected by IP address hits on black lists
- 31935 messages rejected due to invalid recipient
- 32402 messages rejected due to SpamAssassin score of 10 or greater
- 2788 viruses and other malware (most caught by ClamAV)
Individual users might not be seeing the increase because spam-blocking software mitigates the problem to a degree. Still, the solutions aren’t adequate. A lot of spam makes its way past the filters. System administrators are having to spend more time on the problem and they need more servers and bandwidth to deal with the increasing flow of spam.
Of course, today, spam is not limited to email. Trackback and comment spam can hammer a blog server and they are a common reason why bloggers disable such features. I opened Skype today for the first time in a while and I had two XXX messages in an hour, giving a new meaning to “Call Girl.” Nonetheless, let’s stay focused on email.
I emailed a group of people who have been around the Internet a long time to see what they thought of the future of email. I asked them specifically if we’re losing the war on spam.
Brad Templeton
Brad is Chairman of the Board, Electronic Frontier Foundation (EFF).
It seems to me that we’re losing the battle. The
spammers have won and there aren’t any solutions in sight.I wouldn’t say that. There are a number of fairly decently working
filtering systems, though a number of them have concerns about false
positives. This doesn’t rely on draconian blacklists, though some people
use them.There are a number of techniques not yet tried.
There are areas where we’re losing, namely in the botnet department.
As long as so many people run insecure systems, we are going to have
botnets, and they will deliver spam that’s hard to deal with except
by filtering and challenge/response.
Paul Vixie
Paul is the author of several RFCs and founded MAPS (the mail abuse prevention system), known for its real-time blackhole list.
Is this a war we can win?
not with smtp.
Certainly, we’re not winning it now.
right.
every potential smtp improvement or replacement that could do anything to
actually stop spam, has been systematically patented. the crap that’s left
isn’t going to do any good. we’re headed for walled gardens.
Eric Allman
Eric is one of the authors of the SMTP RFC and the developer of Sendmail.
Are we losing the war on spam?
It depends on how you define “win”. I still get junk phone calls, but the phone system is reasonably usable today. I think that spam can get to that level.
Is the war on spam winnable?
By the definition above, yes, but not without cost. As you probably know, I’ve been working on DKIM for cryptographic signatures on email. Assuming that DKIM is accepted and deployed, we’ll be able to invert our way of thinking to make it more like the real world — and more like IM.
In the real world I don’t let anyone walk into my house. I look through the peephole to decide if they are someone I know or expect first. Right now we let just any old piece of email walk into our houses. Similarly, IM uses buddy lists, and it’s not uncommon to only accept messages from buddies. Both of these cases are “filter in” vs “filter out”. Right now we filter out messages that we consider to be spam, and everything left is treated as good mail. In the future I think we’ll see a much more nuanced approach. Because of the nature of email it won’t be pure “filter in”, but rather something like this:
(1) Am I sure who sent the message (i.e., did the DKIM signature verify)?
NO: go to step (4)(2) Do I know and trust the sender?
YES: accept the message(3) Do my peers know and trust the sender?
TRUST SENDER: accept the message
KNOWN BAD GUY: refuse or drop the message
UNCLEAR: continue to step (5)(4) Does the purported sender sign all messages?
YES: must be a forgery; refuse or drop the message(5) Content scan the message — is it probably spam?
ABSOLUTELY: refuse or drop the message
ABSOLUTELY NOT: accept the message
NOT SURE: quarantine the messageThis is over simplistic, but essentially all we are doing today is step 5, and even then we usually err on the side of accepting the message (that is, if in doubt, accept it) in order to avoid false positives. But let’s imagine a day when 80% of my incoming mail is signed. That means that less than 20% gets to step 5 (since step 4 also culls out some messages), and I can probably afford to turn up the sensitivity on my spam filters (if in doubt, don’t accept it) without making my false positive rate work. This analysis is horribly over-simplified, of course, but the point is that we will be able to do a better job in the future than we can today.
Is there any reason for hope?As described above, yes.
David Strom
David is a long-time writer on networking and email, including his own newsletter, The Web Informant at www.strominator.com.
I think the war is pretty much lost. Yes, the volume of total spam is vast compared to real message traffic, and won’t be going down anytime soon.
You have to have spam blockers at various places on your network just to survive — a gateway appliance, filters on each email inbox, and let’s not forget about AV tool. It is a constant battle of wits, and an arms race as the blockers try to stay one step behind the spammers.
All in all, very depressing. I don’t think the war is winnable unless we move towards sender authentication or secure email, which for the most part people are opposed to do.
Danny Goodman
Danny is an author of many books including SpamWars. On his spamwars.com site, he reports today that 96.3% of yesterday’s email was “unwanted.”
Danny and I spoke by phone. He said: “It’s a lot like the war on terrorism. The hardest part is defining what the war is. The offenders are not clearly defined, the war is not clearly defined.” He said the war seems like “a constant game of whack-a-mole.”
In our conversation, we discussed that there were three approaches to combat spam: legislation, technical and user education.
“Legislation is weak, and in some examples, it almost legalizes some forms of spam. Enforcement is next to impossible. Plus, the amount of money to put a case together is incredible.” He wasn’t too optimistic that technological solutions would be acceptable. We discussed that the ability to use cryptographic signatures on email has been around (PGP) but it has not been widely adopted. “This is so obscure to most people,” he said.
We discussed sender-verification in which an email server that receives an email contacts the sender to verify that its server sent the mail. When I discussed this option with Bob at O’Reilly, he used to do that but the volume of email (i.e., the volume of spam) makes that impractical. His already overburdened servers would have twice the workload. I am surprised that the technical community has not come up with a technological solution.
Danny mentioned a new form of spam that’s been popping up: image spam. The content of the spammer’s message is contained within an image to get by content filters. In response, spam-filtering companies are starting to use OCRs to detect words in images. And, in response to that response, spammers begin to distort images slightly so that they can’t be read accurately by OCR software. The war escalates, perhaps making the point that spam-blocking solutions don’t truly eliminate spam.
Danny discussed bots, which are installed on insecure computers around the globe, and do a bulk of the spam. I asked him if perhaps we should combat spam by writing programs that go out and remove bots. He said: “It’s kind of happening but not in a good way. Rival gangs of bot developers are writing code to remove bots installed by other gangs and install their own.”
Danny believes that user education might make a difference. He’d like to have Oprah do a show on spam to educate users so that they don’t respond to spam and don’t operate computers that are vulnerable to bots. I’m skeptical that you can educate people to not respond, especially when spammers are so skilled at deception. I don’t know how to educate my own kids about what to do when I’m often confused by pop-ups that are disguised as messages from the operating system.
“In theory, the war could be winnable,” said Danny. If nobody responded, spammers would go away. “That there is sufficient response as small as it may be manages to feed the spam economy. ”
Danny’s bottom line is this: “Each email recipient must be suspicious of every piece of email that arrives in his email box.”
The Decline and Fall of Email?
Can email be saved? Are the days for SMTP numbered? Is this most basic Internet application so badly designed for the kind of online world we now inhabit that maybe we should think of leaving it behind? Is anybody working on the problem?
USENET was once a great Internet service. As it became more popular, more and more spam was flowing through it. Soon users moved on, finding other ways to communicate that didn’t have the same problems, at least not yet. I do wonder if we could actually trace a migration from one set of services to the next, based on users leaving a sufficiently polluted space for greener territory.
Every Internet application that demonstrates the value of collective intelligence is eventually met with sophisticated attempts to dump garbage. Will any sufficiently open social network be met by ever more anti-social behavior until it eventually collapses?
The barbarians are at the gateway.