Mon

Mar 5
2007

Dale Dougherty

Dale Dougherty

Another War We're Not Winning: Us vs Spam

Are we losing the war on spam? Is the war on spam a war we can win? Is there any reason for hope?

When I learned how much spam was hitting our servers at O'Reilly, I decided to ask several long-time Internet luminaries these questions. Was the situation as bad as I thought it might be? In short, the answer is yes, which only makes me wonder why more people aren't talking about it.

Let's first try to quantify the problem using O'Reilly's servers as an example. I'd like to see how we compare to other organizations.

All of our incoming email goes through one of two gateways, which route mail to servers that decide to accept or reject the message. This is, of course, before the message is delivered to an O'Reilly user, who may apply additional spam filters in their email program. The bottom line, according to Bob Amen, Director of Systems Engineering at O'Reilly, nearly 95% of ALL incoming messages are spam.

Here are the amazing stats that Bob shared with me. These numbers represent a one-week snapshot (last Monday 2/26 to Sunday 3/4.)

  • 829,890 SMTP connections made to our two gateway mail servers
  • 904060 attempted message deliveries
  • 49194 messages accepted (I think this is actually a little high due to a configuration problem with our Zimbra server.)
  • 94.6% of all messages were rejected
Here's the breakdown on how we reject messages:
  • 282414 connections rejected due to bad SMTP HELO syntax
  • 224722 connections rejected by IP address hits on black lists
  • 31935 messages rejected due to invalid recipient
  • 32402 messages rejected due to SpamAssassin score of 10 or greater
  • 2788 viruses and other malware (most caught by ClamAV)

Individual users might not be seeing the increase because spam-blocking software mitigates the problem to a degree. Still, the solutions aren't adequate. A lot of spam makes its way past the filters. System administrators are having to spend more time on the problem and they need more servers and bandwidth to deal with the increasing flow of spam.

Of course, today, spam is not limited to email. Trackback and comment spam can hammer a blog server and they are a common reason why bloggers disable such features. I opened Skype today for the first time in a while and I had two XXX messages in an hour, giving a new meaning to "Call Girl." Nonetheless, let's stay focused on email.

I emailed a group of people who have been around the Internet a long time to see what they thought of the future of email. I asked them specifically if we're losing the war on spam.

Brad Templeton

Brad is Chairman of the Board, Electronic Frontier Foundation (EFF).

It seems to me that we're losing the battle. The spammers have won and there aren't any solutions in sight.

I wouldn't say that. There are a number of fairly decently working filtering systems, though a number of them have concerns about false positives. This doesn't rely on draconian blacklists, though some people use them.

There are a number of techniques not yet tried.

There are areas where we're losing, namely in the botnet department. As long as so many people run insecure systems, we are going to have botnets, and they will deliver spam that's hard to deal with except by filtering and challenge/response.

Paul Vixie

Paul is the author of several RFCs and founded MAPS (the mail abuse prevention system), known for its real-time blackhole list.

Is this a war we can win?

not with smtp.

Certainly, we're not winning it now.

right.

every potential smtp improvement or replacement that could do anything to actually stop spam, has been systematically patented. the crap that's left isn't going to do any good. we're headed for walled gardens.

Eric Allman

Eric is one of the authors of the SMTP RFC and the developer of Sendmail.

Are we losing the war on spam?

It depends on how you define "win". I still get junk phone calls, but the phone system is reasonably usable today. I think that spam can get to that level.

Is the war on spam winnable?

By the definition above, yes, but not without cost. As you probably know, I've been working on DKIM for cryptographic signatures on email. Assuming that DKIM is accepted and deployed, we'll be able to invert our way of thinking to make it more like the real world --- and more like IM.

In the real world I don't let anyone walk into my house. I look through the peephole to decide if they are someone I know or expect first. Right now we let just any old piece of email walk into our houses. Similarly, IM uses buddy lists, and it's not uncommon to only accept messages from buddies. Both of these cases are "filter in" vs "filter out". Right now we filter out messages that we consider to be spam, and everything left is treated as good mail. In the future I think we'll see a much more nuanced approach. Because of the nature of email it won't be pure "filter in", but rather something like this:

(1) Am I sure who sent the message (i.e., did the DKIM signature verify)?
NO: go to step (4)

(2) Do I know and trust the sender?
YES: accept the message

(3) Do my peers know and trust the sender?
TRUST SENDER: accept the message
KNOWN BAD GUY: refuse or drop the message
UNCLEAR: continue to step (5)

(4) Does the purported sender sign all messages?
YES: must be a forgery; refuse or drop the message

(5) Content scan the message --- is it probably spam?
ABSOLUTELY: refuse or drop the message
ABSOLUTELY NOT: accept the message
NOT SURE: quarantine the message

This is over simplistic, but essentially all we are doing today is step 5, and even then we usually err on the side of accepting the message (that is, if in doubt, accept it) in order to avoid false positives. But let's imagine a day when 80% of my incoming mail is signed. That means that less than 20% gets to step 5 (since step 4 also culls out some messages), and I can probably afford to turn up the sensitivity on my spam filters (if in doubt, don't accept it) without making my false positive rate work. This analysis is horribly over-simplified, of course, but the point is that we will be able to do a better job in the future than we can today.

Is there any reason for hope?

As described above, yes.

David Strom

David is a long-time writer on networking and email, including his own newsletter, The Web Informant at www.strominator.com.

I think the war is pretty much lost. Yes, the volume of total spam is vast compared to real message traffic, and won't be going down anytime soon.

You have to have spam blockers at various places on your network just to survive -- a gateway appliance, filters on each email inbox, and let's not forget about AV tool. It is a constant battle of wits, and an arms race as the blockers try to stay one step behind the spammers.

All in all, very depressing. I don't think the war is winnable unless we move towards sender authentication or secure email, which for the most part people are opposed to do.

Danny Goodman

Danny is an author of many books including SpamWars. On his spamwars.com site, he reports today that 96.3% of yesterday's email was "unwanted."

Danny and I spoke by phone. He said: "It's a lot like the war on terrorism. The hardest part is defining what the war is. The offenders are not clearly defined, the war is not clearly defined." He said the war seems like "a constant game of whack-a-mole."

In our conversation, we discussed that there were three approaches to combat spam: legislation, technical and user education.

"Legislation is weak, and in some examples, it almost legalizes some forms of spam. Enforcement is next to impossible. Plus, the amount of money to put a case together is incredible." He wasn't too optimistic that technological solutions would be acceptable. We discussed that the ability to use cryptographic signatures on email has been around (PGP) but it has not been widely adopted. "This is so obscure to most people," he said.

We discussed sender-verification in which an email server that receives an email contacts the sender to verify that its server sent the mail. When I discussed this option with Bob at O'Reilly, he used to do that but the volume of email (i.e., the volume of spam) makes that impractical. His already overburdened servers would have twice the workload. I am surprised that the technical community has not come up with a technological solution.

Danny mentioned a new form of spam that's been popping up: image spam. The content of the spammer's message is contained within an image to get by content filters. In response, spam-filtering companies are starting to use OCRs to detect words in images. And, in response to that response, spammers begin to distort images slightly so that they can't be read accurately by OCR software. The war escalates, perhaps making the point that spam-blocking solutions don't truly eliminate spam.

Danny discussed bots, which are installed on insecure computers around the globe, and do a bulk of the spam. I asked him if perhaps we should combat spam by writing programs that go out and remove bots. He said: "It's kind of happening but not in a good way. Rival gangs of bot developers are writing code to remove bots installed by other gangs and install their own."

Danny believes that user education might make a difference. He'd like to have Oprah do a show on spam to educate users so that they don't respond to spam and don't operate computers that are vulnerable to bots. I'm skeptical that you can educate people to not respond, especially when spammers are so skilled at deception. I don't know how to educate my own kids about what to do when I'm often confused by pop-ups that are disguised as messages from the operating system.

"In theory, the war could be winnable," said Danny. If nobody responded, spammers would go away. "That there is sufficient response as small as it may be manages to feed the spam economy. "

Danny's bottom line is this: "Each email recipient must be suspicious of every piece of email that arrives in his email box."

The Decline and Fall of Email?

Can email be saved? Are the days for SMTP numbered? Is this most basic Internet application so badly designed for the kind of online world we now inhabit that maybe we should think of leaving it behind? Is anybody working on the problem?

USENET was once a great Internet service. As it became more popular, more and more spam was flowing through it. Soon users moved on, finding other ways to communicate that didn't have the same problems, at least not yet. I do wonder if we could actually trace a migration from one set of services to the next, based on users leaving a sufficiently polluted space for greener territory.

Every Internet application that demonstrates the value of collective intelligence is eventually met with sophisticated attempts to dump garbage. Will any sufficiently open social network be met by ever more anti-social behavior until it eventually collapses?

The barbarians are at the gateway.

tags: email  | comments: 29   | Sphere It
submit:

 
Previous  |  Next

0 TrackBacks

TrackBack URL for this entry: http://blogs.oreilly.com/cgi-bin/mt/mt-t.cgi/5302

Comments: 29

  Paul Annesley [03.06.07 12:31 AM]

Project Honey Pot at http://www.projecthoneypot.org/ is an interesting looking project, aiming to gather data about email harvesters by linking unique generated "honeypot" email addresses to the IP addresses they were served to, and then tracking spam received.

Perhaps targeting the problem from this end will make more of a difference than just more and more inbound mail filtering.

  Alan [03.06.07 12:49 AM]

how about having a service on the MX record holding servers that has a list of authoritive IP's that can send email on its behalf.

smtp relays will always exist, but if a process was in place that allowed them to be slightly more trusted, that would be cool.

if this was a RFC, it could ensure worldwide use.

also, if systems don't support this, spam solutions could provide a feature to add points to the message spam score.

  Ben [03.06.07 01:23 AM]

Alen, you're thinking of Sender Policy Framework (SPF) its not in the MX record but it is in the DNS, it says what servers are aloud to send mail on its behalf.

It won't stop spam, in fact spammers often use SPF to verify their E-mail adresses, but it will mean that spam cannot come from hotmail or similar without being sent from the hotmail servers.
http://en.wikipedia.org/wiki/Sender_Policy_Framework

Personally I think that the war on spam is really the war on botnets, you remove the botnets you remove most spam.

  I am dumb [03.06.07 01:29 AM]

If you can't block it, reply to it! And fool them. No? Mechanisms of evolution (Darwin)?

  Frank LaRosa [03.06.07 01:30 AM]

It should cost money to send an email message. People need to stop resisting this idea and start working on it - it's by far the most effective way.

Email should cost a nickel a message. This tiny amount would not burden individuals or legitimate businesses, but it would completely eliminate spam.

When you consider the costs we're already paying to use email - not just what we pay directly for spam filtering services but also the wasted bandwidth and millions of hours of lost time spent sifting through spam - spending a few cents per message is a bargain.

  Rick [03.06.07 01:47 AM]

The war on spam can easily be won if there is the political will to do something about it. Those that profit from spam can easily be traced. The bots that send spam can easily be traced and taken down.


The only problem is that nobody is doing anything about it, which in turn is caused by the illusion that we could deal with it with technological means.


The only thing we've succeeded in with technological measures is to hide the problem from the general public and the politicians. Nobody outside our little techno-geek community considers spam a major problem because they are hardly confronted with it.


I say, open the floodgates, switch off the DNS-blocks and content-filters and show the rest of the world what the real problem is, if only just for 48 hours.


Before they actually start listening to people who basically want to kill off e-mail by adding authentication mechanisms or charging money per e-mail.


(I get so pissed of when I read crap like "Email should cost a nickel a message. This tiny amount would not burden individuals or legitimate businesses, but it would completely eliminate spam." No, it will just destroy entire non-profit communities, but hey, as long as we can still do business, who cares? That's exactly the kind of egocentrical attitude spammers have. Thanks a lot, [self-censorship applied].)

  Ron [03.06.07 02:23 AM]

A few years ago already, I remember reading someone's suggestion that all outgoing e-mail be required to have a "stamp" which amounted to nothing more than a delay of x number of seconds between messages from a single source. Your average email user would never notice such a delay because of the low volume. Legitimate companies could "register" (for a fee or "whatever") to have the delay removed and would be immediately blacklisted if they violated the rules. Is something like this feasible given today's environment and technology? If enough of the primary routers in the world were programmed to do this, would it help?

  I am dumb [03.06.07 02:28 AM]

Filtering make spamming more difficult. And this is like the Maginot Line (France), the spammmers will try to avoid it.
To give (long-term) peace a chance, we have to make spamming (war) uninteresting for all. Being poor value (for money), etc.

  Justin Mason [03.06.07 03:57 AM]

A pretty good overview of the current situation, I think. Some comments:


1. complaining about continual spam escalation misses the point -- anti-spam and anti-abuse is adversarial. It will _always_ require continual, ongoing work, and will _always_ keep escalating. That is the essential nature of spam, IMO.

2. having said that: IMO, botnets are the biggest problem right now. The reason the SMTP infrastructure is falling over is because of the massive amplification botnets allow. My own personal SMTP server fell over under extreme load a few months back -- even _without_ any kind of CPU-intensive filtering active! It quite simply could not accept SMTP connections fast enough to keep up with the spam traffic.

That was 3 months ago; since then, the volume levels have continued to rise steadily. That immense volume bodes ill for the future of _any_ kind of filtering.

3. Bigger receivers, like ISPs, have a much worse situation to deal with than we do; I can spare CPU time to perform decent filtering, but they tend to skip the heavy crunching in order to support thousands of customers on limited hardware -- generally by rejecting during the SMTP transaction with the more unreliable blocklists like SORBS, Spamcop, or even the totally inappropriate SPEWS. This results in a huge false positive rate, which because it rejects traffic before it's delivered, is never even noticed by the recipient unless the sender finds a way to contact them out-of-band. That's a lousy situation.


4. sender-address verification is _bad_ (I must write a post about this over at taint.org). Basically, there's one obvious response for spammers looking to evade it -- use "real" sender addresses. Where's an easy place to find real addresses? On the list of target addresses they're spamming! Hence, the spam recipients now get twice as much mail from each spam run -- spam aimed at them, *and* bounce blowback from hundreds of spams aimed at others. It's the obvious response to SAV, which is one reason why we never implemented something like that in SpamAssassin.

5. Legislative/legal responses in the US have been a woeful failure. This is a shame, as at one stage this was a practical way to deal with the problem; it's probably gone way too far now, with the serious criminals now involved, for this to help any more.

  Stephen Lord [03.06.07 04:43 AM]

Yep well any "war" on a non existant entity is doomed to failure.
As noted above the problem is lack of any enforcement. We know who the spammers are, we know that they can hide in countries not overly concerned by this and we know that companies in our own countries are using services to SPAM.

What needs to happen is personal, corporate and state responsibility but this will cost money.
Who is going to pay? Non profit organisations or individuals? The state? or the commerical users?

We have 1001 different solutions, non working with any of the other 1000 and most of them not adressing the root of the problem which is companies and people make money from Spam!

The technology exists and is easily implementable but it can only work if legitimate mails are somehow allowed and the only way I can think to do this is by requiring registration and accountability for bulk mails.

There are lots of legitimate bulk mails, from artists who want to publicise an event to people who have signed up for this to webmasters wishing to tell subscribers that the site will be down for maintainance for some hours etc. I personally do both of these but the criteria are different.

The main problem with most SPAM blocking/filtering is simply it fails to take into account the different types of email. Its doomed to failure or penalising the non profits.

For artists 90%+ of the mail is probably local, be it a band or a painter ... they don't need delivery outside of the local area except for a few cases ... herein lies a solution to the filtering approach. The majority of the bulk mails can be filtered based on source/destination and the emailer subscribe to a free service. Emails needing to be delivered outside the local area would then need to be sent as individual mails. The total emails and the ones outside of the subscription can eaily be monitored.

For international websites a different approach is needed .. but again a subscription is simple and gives an appointed authority the ability to monitor. For countires not subscribing ... they are simply blocked, personally I won't miss another Nigerian eMail ever.

This effectively also prevents open relays, the open relay is useless if the person is not subscribed to bulk mailing.

Finally who pays? Well the nickel a email sounds bad and will penalise non profit org's so why not have a system based on emails offering products for tangible price, just 1-2% of the product price.

I get bulk eMails from airlines, train companies etc. I don't massively mind but if they had to pay a small amount per email based on the price of the offer the more mindless ones would dissapear, just because I took a Australian domestic carrier when I was in Australia doesn't mean I want their publicity. They would be forced to target these emails soley to prospectively interested people... or the ROR would not be worth it.
For emails about FREE upgrade etc. this should then be based on the price of the original product.

Finally where does SPAM come from?
A significant amount comes from private servers... some from webmail type accounts etc. and a lot from open relays.
Why not start making people individually or corporately responsible.
I frequently get DDOS and cracking attempts, in most cases I trace back and email the relevant responsible but in probably 80% of cases get no reply. Whereby this makes it easier for me to ban the originating IP it doesn't reduce the number of emails they send. Why not define in each subscribing country that emauils to abuse@ MUST be replied to or they loose the right to send email ?
Yes this costs money... but so does SPAM... and the people it costs most is industry so they should pay because they will benefit from it most.

  Konrad Foerstner [03.06.07 05:00 AM]

I am not sure if fighting SPAM alone makes the world a better place. SPAM is a symptom that shows that (1) there are too many unsecured computers connected to the net and (2) there is enough criminal energy out there that makes use of them. If the SPAM problem is solved somehow (some proposals are mentioned above) both points aren't solved. It might lead to an increase of other computer crimes like blackmailing companies with DDOS attacks using botnets. Crime is partly the result of missing options of legal work for the people committing the crimes, but well ... that's hard to solve. Increasing general security of computers is easier. Unfortunately the focus in most companies/communities seems to be on implementing as much features as possible while increasing complexity and reducing security.

  Justin Mason [03.06.07 05:20 AM]

BTW I've followed up on my own blog here: http://taint.org/2007/03/06/141708a.html

  Anonymous [03.06.07 07:02 AM]

It would seem to me that the although spam is a problem all by itself, it has really been exacerbated with the advent of botnets. And botnets are relevent not only to spam, but to all areas of computer networking and internet protocols. Solving the larger botnet problem should go a long way to solving (or at least mitigating) the spam problem, by removing its most potent delivery mechanism.

So, shouldn't we be focusing on botnets rather than spam?

As for how to solve the problem of botnets? I don't know. But I know it will not be solved by continuing the plug-all-the-holes game that is the current state of computer security. There must be more fundamental, architectural changes to how computer networks work, and to how software relates to the operating system on which it runs.

I might make a few people angry with my last comment here - but I think that things like spam, botnets, and viruses are actually a good thing. Although they are annoying and costly, they do serve to point out flaws in the design of our internet system. This pushes us to rethink, redesign, and strengthen our internet system (on which our economy now depends).

  Eric Meyer [03.06.07 07:53 AM]

The irony of this message, at least for me, is that any e-mail I send from my work account to the O'Reilly servers gets bounced as being spam, because your blacklist apparently doesn't like 'theplanet.com'.

I know of several authors whose servers are colocated at The Planet, as it happens, so I guess none of them can get through either. Oh well.

  Greg [03.06.07 07:55 AM]

This can't be won with SMTP but SMTP can be used as the foundation for the next generation email and solution.

I have a concept for Email 2.0 that you may find interesting:

http://www.raizlabs.com/blog/?p=88

  Henrik [03.06.07 09:30 AM]

The best technical suggestion I have heard about so far is stub mail. The proposed protocols are a bit daft, but the priciple is sound. "Sender hosts the email, and merely notifies the recipient". Yes it will require a change to all email-client software, but I think an upgrade path using smtp is possible, which is key as the transition would take time.

  Michael R. Bernstein [03.06.07 09:43 AM]

I found Paul Vixie's reply suggestive.

I suspect that once email hits a tipping point of complete unusability, the US government will mandate an involuntary patent pool of some kind, similar to what was done in 1917 with aircraft patents, thus removing the barrier posed for years by the Wright and Curtiss patents.

Given the disaster that Tamiflu turned out to be, something similar could also happen within ten years (if we suffer through a couple of pandemics) with certain basic biotech patents.

  John Wesley [03.06.07 09:53 AM]

Interesting conversation. As someone who is more of an ordinary user than a server admin I haven't really noticed an enormous amount of spam. Some yes, but it gets caught by my spam filter.

From the way it sounds this is a much bigger problem than it appears. Making the general public more aware would be a big step.

  Rulf [03.06.07 10:11 AM]

Why not make pay Microsoft for cleaning up the spam mess which is almost exclusivelly due to their lousy Windows OS??? Without the gross incompetence of Gates and his Redmond gang there wouldn't be botnets.

  Valentin [03.06.07 10:25 AM]

I think one way forward that is missing in the discussion is: legislation to reduce the size of botnets.

Make it mandatory that PCs sold with OS installed also have a virus scanner with a five year license. That everyone offering a mail service must have server side scanning for viruses, that ISPs must (per default) protect their costumers with firewalls and have a look at outgoing smtp connections. - yes, this could make PC's, email accounts and internet connections more expensive - but the cost for dealing with SPAM, DDOS and other kinds of cybercrime would decrease.

I have described this idea in more detail a while ago: Public Health and Cybercrime

  Matt Sergeant [03.06.07 10:39 AM]

Wow, you sure got some biased replies there.

No SMTP's not dying. Yes we have to continually re-evaluate email but then everyone used to run open relays too - now they don't. Things change.

The only real worry is the sheer volume of traffic people are having to deal with, but new technologies and changes in SMTP software are helping to mitigate that.

I'm not saying the problem isn't bad, but it's far from time to throw our hands up in the air and declare email dead. That way lies the madness of trying to re-invent email.

It's worth remembering to look at the situation in your inbox. If your inbox is full of spam then you probably need a better filtering solution, not to cry about email being hopeless. I have a domain that gets absolutely hammered with spam (2 users, 30k spams a day) and yet my inbox has only one or two spams a day. Frankly I get more snail-mail junk.

  David Troup [03.06.07 01:08 PM]

Part of the problem is that spam continues to evolve, but most anti-spam engines haven't.

We take the approach of re-engineering our engine every few years and its make a HUGE impact on our ability to deal with the new nasties like image spam, etc.

Learning based systems with "scoring" are things we DON'T use because they guess at spam which leads to lost emails (false positives).

Spam made email a pain in the butt :

Anti-spam systems made email a non-trusted communications conduit because of the uncertainty it has introduced into the system.

Our goal is to change that. So we built our latest anti-spam engine from the ground up to protect the message at all costs. Killing spam isn't hard (for us), its not killing your legitimate email that seems to be a big issue for the anti-spam industry.

I didn't want this to come across as a sales rant - sorry if it did. I wanted to point out the different approach we've taken towards email filtering.

David

  Anonymous [03.06.07 08:17 PM]

Danny Goodman comes closest with "user education". Nearly everyone is thinking about the wrong end of the problem: the spammer, not the willing spamee.


To abolish spam:



  1. ISP changes its Terms of Use to give fair warning.
  2. ISP sends its own customers fake spam.
  3. Every customer who responds favorably to the fake spam finds his account suspended.
  4. Even the most spam-friendly user learns to shun spam, or can't use the Internet.
  5. Spamming no longer makes money. The spammers give up.

Advantages over the other proposed solutions:

  • This one will work
  • No heavy-handed legislation needed
  • Leaves anyone free to run a mail server at least as easily as they can now
  • No one has to go to jail, pay huge fines or be censored
  • Implementation can start immediately.

It's all up to the ISPs. Do they want to stop spam?

  Alan 2 [03.07.07 03:37 AM]

Henrik's right: IM2000 or “stub mail” is the way of the future.

In the meantime, a probabilistic engine like DSPAM or CRM114 will get you 99.5%–99.95% accuracy.

  Rob Mueller [03.07.07 03:55 PM]

There's one simple solution that would basically solve the current spam problem.

Block all port 25 connections from dialup/adsl hosts

A lot of ISPs are already doing this, and it's a good thing because it forces users to either use the ISPs email server via an authenticated SMTP session, or a 3rd party trusted server via an alternate SMTP port (either SSL one, or the "mail submission" port, or some arbitrary other one).

This simple act completely changes the skew of the whole problem.

  • Botnets are immediately blocked from sending spam directly to other email systems
  • This means that spammers either have to setup their own systems, which have very limited IP sets and are blocked easily, or they have to change the way their botnets work

Assuming they change the way their botnets work, they have a couple of options

  • Hijack user accounts to try and send via the ISPs SMTP. In this cases, the ISPs would quickly be able to detect users sending the spam, and block their machines completely and inform them that they have a virus on their machine. We kill two birds with one stone.
  • Start signing up and using free webmail type accounts to send spam (we're already seeing this). Then it's up to the webmail providers to detect and stop this. The poorly run ones will end up on block lists with a poor reputation, the smart ones solve the problem

That would vastly, vastly reduce the spam problem almost immediately. The real problem is that ISPs don't want to block port 25, because suddenly then something that wasn't their problem, does become their problem. On the other hand, the amount of bandwidth ISPs must be loosing due to botnets must be huge, so you would think it would be in their interest to do it.

  davidm [03.08.07 07:52 AM]

Automated systems and those that rely on networks of trust give me the creeps. I don't want to close out someone just because I don't know them, and either people I know don't know them or they don't like them. Talk about living in a walled garden. The openness that is biting us is also a big plus of the net as it is today.

Looking at Eric Allman's message, it'd be worthwhile to simply add a "turing test" - is there a real person sending the message, what do they want - which would be part of step 5. That would eliminate 98% of current spam and still let people reach you.

Otherwise, I agree with Rick. I want to find a better solution than treating the openness of the net as a problem.

  Steven Champeon [03.08.07 04:07 PM]

There are many wars on spam, just as there are many wars on different types of cancers. You can break them down into several different styles of attack and defense and escalation and withdrawal, as well. But they're all simply forms of abuse, which we've allowed to happen and flourish due to several factors (lack of authentication in SMTP, poor security in Windows, a complete lack of domain registration/whois/rwhois oversight by ICANN, an utter failure on the part of many ISPs to police their own networks, and so forth).

  • so-called "mainsleaze" spam, where well-intentioned but bumbling or badly socialized corporations and their representatives disrespect your consent; we will eventually fix most of this through end user education and reputation services.
  • botnets; the biggest problem, because solving it relies on a secure computing environment that we may never have, or on ISPs having the guts to secure their own networks and protect the rest of us from their infected users. Fortunately, rather easy to detect if you have a sufficient set of data about hosts with generic, provider-assigned reverse DNS.
  • 419/lotto/Spanish Prisoner scams; a few good hangings would help here, but it shows both the international character of spam and the pathetic state of account setup vetting at many freemail providers. Could disappear if better checks on freemail accounts were instituted, but hard to say whether that's economically/politically feasible.
  • pump and dump scams; if the SEC didn't tolerate them, there wouldn't be any of them. Hopefully, we're seeing the first steps towards reducing the success rate of this type of scam.
  • blowback; it's still amazing to me that because someone decides to send a few million spam messages with a forged sender or multiple forged sender addresses purporting to come from my domain, I have to buy new hardware just to deal with the extra load from bounces I should never have received in the first place. Fix that by suing the antispam appliance vendors, first, then blacklisting the remaining offenders. It's insane and fundamentally irresponsible.
  • other media: blog/trackback/comment spam; this is trickier to deal with because whereas you can say with relative ease whether an IP is a legitimate source of mail, it's very difficult to come up with useful Venn diagrams of "the Web browsing public". Fortunately, this can be fought with URL/domain-oriented blacklists like SURBL and URIBL, since most of these guys are just trying to boost their PageRank or get you to surf to their casino/pillshop/sextoy outlet.

The bottom line is that you don't "fight spam", you fight abuse of a variety of different kinds and degrees, with the tools and tactics appropriate to each kind of abuse. And if you do it right, you take the battle to the perimeter, instead of simply letting the user eat everything. I'm stunned into slackjawed silence by the amount of time wasted on developing and tweaking end user spam filters, when it's relatively easy to detect most of the spam with a few simple rules, and when you combine them, to use scoring mechanisms to allow end user policy rejection at the edge, rather than hoping that the user's "Spam Folder" will catch all of the effluvia you've let in.

  Frank Wiles [03.12.07 10:12 AM]

You talk about having too many incoming SMTP connections, this was a problem for myself personally and some of the ISPs I consult for. While it is a commercial product, TrafficControl from MailChannels allowed us to go from being able to handle 300 SMTP to 3,000 connections on the same hardware.

It also has a new interesting way to fight off spam, by throttling down the bandwidth to any IP based on RBLs, OS, etc. This works pretty well considering most bots give up after a few seconds if they aren't getting good flow.

  Daniel [03.14.07 08:46 AM]

If possible the US GOV Should just impose a tax on unsolicited digital mail/medi. as long as it excluded business interacting and advertising to their existing customers etc. a few cents per message ads up quick to these spam companies... Is that possible. any thoughts.

Post A Comment:

 (please be patient, comments may take awhile to post)






Type the characters you see in the picture above.

RECOMMENDED FOR YOU

RECENT COMMENTS