When you get more than one thousand spams a week, you live for improvements of a couple of tenths of a percent. I of course upgraded immediately.

It was a little anticlimactic. Given the minor delta between my 99.5% accuracy and John’s 99.9%, I did not see an immediate difference in my spam protection. It remains very, very good. (Surprise. Given the small difference in accuracy, it will take months to have something to compare.)

Tonight I got around to reading the more detailed version history, which explained the improvements in a bit more detail:

Made lots of improvements to SpamSieve’s parsers and tokenizer for better accuracy. To fully take advantage of this, you will need to reset SpamSieve’s corpus and re-train it (e.g. with 300 recent good messages and 600 recent spams). However, this is certainly not required, and I expect that most people will opt for the simpler upgrade of just installing the new SpamSieve application. [Emphasis added.]

Given the amount of spam I get, I want to achieve the maximum level of accuracy, not opt for the easiest upgrade path. So, curious, I checked my Junk mailbox in Eudora: 598 spam messages. A sign from god. I collected 300 recent good messages, backed up my SpamSieve corpus, and reset it. Just finished retraining.

We’ll see how it goes. In a month or two.

So I did a few things about it.

WordPress 1.5 has a number of new features that make dealing with comment spam a little easier, and I’ve adopted those, while also disabling all of the anti-spam plug-ins I had been using in WordPress 1.2. I also recently implemented some suggestions for making it a little harder for automated commenting, adopting a few of the ideas from A short monograph on the theme of blog comment spam, and a couple from Tom Raftery.

I had a problem with the fake form I implemented, because of a misplaced “%”, which caused it to show up and screw up the regular form. That’s fixed, and I think the fake form should be invisible to everyone using a modern browser.

Hard to say how effective it really is at this point, too soon to tell. But at least so far, no spam comments have been left since I implemented the changes and new traps.

But in my own, more personal battles with spam I’ve been more successful at holding back the tide. My stats for 2004:

Filtered Mail
36278 Good Messages
72239 Spam Messages (67%)
197 Spam Messages Per Day

SpamSieve Accuracy
135 False Positives
451 False Negatives (77%)
99.5% Correct

Nearly seventy five thousand spam messages came at me, but thanks to SpamSieve a mere 451 made it into my Inbox. That’s less than two spams a day. Simply amazing.

Even more amazing is that the second half of 2004 was far better than the average for the entire year, meaning that while the rest of the internet went to hell, my own situation actually improved as the year went on:

Filtered Mail
15729 Good Messages
32650 Spam Messages (67%)
176 Spam Messages Per Day

SpamSieve Accuracy
53 False Positives
181 False Negatives (77%)
99.5% Correct

I would attribute this improvement primarily to the new filters I added to my oldest, most spammed email address, and the deactivation of a few temporary addresses I created.

Interestingly and unfortunately, the spam volume seems to have begun to go up again in 2005, as so far I’ve averaged 199 spam messages a day this year. I’ve expired a few more highly targeted addresses, though, in hopes of beating some of it back. Time will tell…

It turned out the issue was a couple of the PHP “open” tags — the text tokens that are used to signal that what follows is PHP code, not HTML code. The Kubrick template (at least as of version 1.2.6) uses the standard “full” tags almost everywhere, but there were a few instances where the developer used the short versions. Since I have short tags disabled on my server, it was causing a parse error, the end result of which was that the comments form could not be displayed. Fixed.

Now that I’ve got comments working, I switched to allowing comments on postings by default, and letting them post without moderation first. We’ll see how that goes.

Because, the weblog world is dealing with a plague of comment spam these days, from “search engine optimization” scumbags who are trying to promote their (or their clients’) websites. So I’ve also installed and enabled a number of comment spam plug-ins. We’ll see how that goes, too, though I’m somewhat pessimistic, for a slew of reasons. Probably I’ll have to turn moderation back on at some point…

And I installed a new caching plug-in, which should theoretically improve the performance of this site. Not really a problem with the volume of visitors I’m getting today, but you never know, and having the plug-in active will hopefully prevent unpleasant surprises.

My very first anti-spam tool was something called Mailfilter. I used it for my personal e-mail on Mac OS X, wrote about it here, and almost immediately afterwards lost a non-spam message to an aggressive keyword match. That was the end of Mailfilter. I can’t even remotely recommend it, as it’s just not intelligent enough (strict, single expression matching), and had zero safety net.

My next attempt at a solution was a utility called SpamFire. Like Mailfilter, it is a “pre-filter,” which means it would run before my e-mail client, download my mail, and skim out the spam. Unlike Mailfilter, it actually saved the trapped messages, so if it made a mistake, I could recover the message. It had plenty of other differences from Mailfilter, which I wrote about previously, and which made it so useful that it became the first anti-spam tool I paid for. But in the end I switched to a different tool because SpamFire was separate from my e-mail client, and that made it cumbersome to use.

In the meantime, I had spam coming into my e-mail at work, and at the recommendation of a co-worker, I installed Cloudmark’s SpamNet (now called SafetyBar), an add-in for Microsoft Outlook. It worked reasonably well, but then they went and started charging money for it. It didn’t work well enough, and I didn’t get enough spam at my work address, for it to be worth paying for, so I stopped using it.

I replaced it with another plug-in for Outlook called SpamBayes. Even more effective than SpamNet, it confirmed for me the value of having a tool that plugs into and works directly with your e-mail client. It made dealing with spam seamless, almost effortless (given the lower volume of spam than my home e-mail addresses). Unlike SpamNet, it’s Open Source, and also unlike SpamNet, it doesn’t depend on a third-party server to run. If I still used Outlook (or had a “work” e-mail address), I would still be using SpamBayes. Effective, easy to use, and free. By far the best anti-spam solution I’ve found for Outlook on Windows. Highly recommended.

My happiness with SpamBayes for Outlook lead me to search for an anti-spam tool for my personal use which integrated fully with my e-mail client, Eudora for Mac OS X. I found Spamnix, which wrapped the Open Source tool SpamAssassin in a Eudora plug-in. I fell instantly in love and dropped SpamFire. This was the second anti-spam tool I paid for.

Spamnix was great for a number of reasons. Its interface within Eudora took the form of a new mailbox, and a couple of simple menu commands. Incoming mail judged likely to be spam was shunted off to the new mailbox; the menu commands let you rescue items that were not spam, etc. It was simplicity itself to use, and fairly effective at trapping spam.

The failing of Spamnix was that it was based on an earlier version of SpamAssassin, which while it had some great regular expressions and other traps for catching spam, it didn’t include any Bayesian filtering at all. This meant that spam filtering was good, but never improved. However, it took a new version of Eudora for me to realize that this mattered. (Spamnix has since been upgraded substantially, and uses the latest SpamAssassin with Bayesian filtering; but I’ve not tried it, I moved on to other things, as you can continue to read.)

SpamWatch was a plug-in shipped with Eudora 6, which I paid for, making it the third anti-spam tool for which I shelled out money. It provided me with Bayesian filtering in my Mac OS X mail environment for the first time. It was a revelation. The first time I downloaded mail after upgrading, SpamWatch caught a half-dozen spam messages, and Spamnix caught several dozen. A quick menu choice fed the missed spam to SpamWatch, and forever afterwards the ratio was reversed. I’ll never again think the same way about learning systems; Bayesian filtering really becomes dramatically more effective the more you use it. I had learned this with SpamBayes at my work e-mail, but I simply didn’t get enough spam at that address for it to really shine.

SpamWatch was so effective that it soon spelled the end for Spamnix. I just didn’t need a second spam filter (and the attendant delay in getting my mail while it processes incoming messages) when the first filter was so good. So I uninstalled Spamnix, and life was good.

SpamWatch actually improved my effectiveness in fighting spam, not just by being a better filter. The new version of Eudora added a new field for mail messages, the Junk Probability, a 1-100 score on how likely a message was judged to be spam. I quickly learned to process my Junk mailbox by first sorting by the Junk Probability, and then scanning for false positives. By sorting first, the messages at the beginning of the list were a lot more in need of review than messages at the end, making it possible to skim more rapidly over items that were certainly spam.

I don’t really remember why I decided to give SpamSieve a try. I do remember noticing that SpamWatch seemed to have hit a plateau, and wasn’t improving any more. I had friends who used and liked SpamSieve, and were getting better stats than I was with SpamWatch, but the interface between SpamSieve and Eudora was, for a long time, through AppleScript, and the integration wasn’t smooth enough. Probably when the version that used a Eudora plug-in came out, I decided what the heck, it won’t cost anything to try it a couple days, and see how it does.

It was not the revelation that SpamWatch was, but it only took a training pass at my archived spam and Inbox (a few thousand messages between them), and a couple of days use proved that it was noticeably better than SpamWatch. After a week I was sold, and opened my wallet, a fourth time, taking my anti-spam expenditures over $100. (Compared to the problem I have with spam, that’s not a lot of dough, money well spent. But it chafes when you realize that the spam problem is entirely due to greedy amoral scumbags who have polluted the e-mail highways and byways to make a few cents per pound of pollutant spread.) I also disabled SpamWatch, since SpamSieve entirely replaced it.

Well, not entirely, not at first. Although SpamSieve had a more sophisticated and accurate Bayesian engine at its core, there was one thing that it didn’t do well, which was assign a Junk Probability to each message. This meant that I lost a very effective tool in my Junk folder processing, of being able to sort the caught messages from least to most likely spam.

I e-mailed Michael Tsai, the developer of SpamSieve, explaining how I used that feature of SpamWatch. He was highly responsive, and agreed that my approach sounded genuinely useful. Unfortunately, at that time SpamSieve’s engine’s algorithms and formulas didn’t really generate a score that could be usefully assigned to Eudora’s Junk Probability column. The math just didn’t work that way. He said he’d look into it, and perhaps in the future, perhaps it might do this.

Fortunately for me, the future did eventually come to pass, and SpamSieve now does assign useful probability scores to processed mail. Its final fault gone, and with the best scoring engine and perfect integration with Eudora, it is by far the best tool I’ve used for managing the amount of spam I get, and I’m happily using it today. Compatible with virtually every e-mail client running on Mac OS X, it totally deserves its reputation as being the best anti-spam tool available on my platform of choice. Highly, highly recommended.

After all that, you’d think I’d be wrapping this posting up. You and I both wish that was true. But no. I’ve got five more tools to write about.

First is Thunderbird, the stand-alone e-mail client that evolved out of the Mozilla suite. It’s available for Mac OS X, Windows, and Linux, and other platforms, too, I think. It’s a nice enough e-mail client, and one of the best for sending and receiving HTML e-mail, if you’re the sort of person who likes that kind of thing.

It has gotten quite a bit better since I evaluated it, to the point where I installed it on Rochelle’s PC, as a replacement for Netscape Communicator, which she had been using since the mid-‘90s. It has a built-in Bayesian spam filter, which works well enough. It’s definitely not as accurate as other Bayesian classifiers I’ve used; SpamBayes and SpamSieve are both quite a bit better, and I think Eudora’s SpamWatch might be a little better. But it’s more than good enough for Rochelle, who gets an order of magnitude fewer spam messages than I do.

Another advantage of Thunderbird is that you’re not using Outlook, which is the number one attack vector for viruses and worms. It’s a good tool, and if you’re stuck on Windows and can’t do better, it’s definitely worth switching to. Recommended.

Another e-mail client I haven’t used much is the bundled OS X e-mail client, Mail.app. It doesn’t work the way I do, and doesn’t seem to be well-suited to managing the volume of mail I get on a daily basis. Supposedly it has good junk mail controls (which use latent semantic analysis), but in the testing I did with it on a secondary e-mail account, it didn’t seem that good, somewhere around the Spamnix level of accuracy. That is, very good, but not excellent. If you like Mail.app, since you can use SpamSieve with it, that’s what I’d recommend doing, and blow off the built-in junk mail controls.

Finally I come to the server-based tools. I’ve saved these for last, not because I tried them last, but because most people won’t be able to use them themselves. Unless you run your own mail server, most of these are impossible to use.

The first server-side solution I tried, and used for quite some time, was what are known as RBL lists. RBL stands for “Real-time Blackhole List”, and the way it works is as mail is received, the mail server sending the message to your server is looked up on a list, which contains known-bad e-mail servers (ones that are known to send spam). If the foreign server is on the list, the e-mail is rejected, in real-time.

In theory this should be extremely effective, and indeed it does cut down on spam considerably. But not completely, and not without collateral damage. There are just too many servers out there that are sending spam, most of them home PCs that have been infected with a worm or virus that converted it to a “zombie” that sends out millions of spam e-mails. It’s impossible to keep the lists current.

It’s also impossible to keep them accurate. There’s no way to maintain the lists with perfect accuracy, mistakes are inevitable, due to both ignorance and malice on the part of people submitting candidates. In the end I had too many messages rejected that were OK, and I had to turn the RBLs off. (I may turn them on again, I go back and forth on the “damn the consequences” philosophy…)

I also tried MIMEDefang, which was a wrapper for SpamAssassin and Vipol’s Razor, with a plug-in for my mail server, Sendmail. I ended up losing mail to this solution, when it would generate errors under load. The failure was intermitent, and impossible to reproduce on demand. Since the mail would just get dropped on the floor, completely lost, I decided I couldn’t afford to try to track down the issue, and simply uninstalled all of it. I’ll surely try another wrapper for SpamAssassin at some point, when I’ve converted my mail server to Postfix, for which it is easier to write plug-ins, and should therefore be easier to write reliable plug-ins.

The last server-side tool I’ve been using doesn’t live on my server. Maybe that’s why it’s worked so well. I wrote about the amount of spam that was coming to my oldest e-mail address at pobox.com, and how the new filters they rolled out saved the address from deletion. I’m still really happy with the results of that service, and plan to keep my pobox.com address for the foreseeable future. Recommended.

In spite of these numbers, I have two things that give me hope.

First, SpamSieve is an amazing anti-spam filter that integrates well with Eudora. It’s far more reliable than the built-in SpamWatch feature that debuted in Eudora 6, primarily in the area of false positives (real messages mistakenly filtered out):

Filtered Mail

13565 Good Messages
22255 Spam Messages (62%)

SpamSieve Accuracy

21 False Positives
197 False Negatives (90%)
99.4% Correct

SpamSieve is award-winning software for Mac OS X, and it integrates beautifully with both Eudora and Mailsmith, the two best e-mail clients for the platform. I am getting to the point where I trust SpamSieve enough to just purge filtered e-mail without reviewing it.

Without SpamSieve, I would be going insane because of spam.

The second thing I have on my side is that more than half of my spam comes to one e-mail address, the oldest e-mail address I still use. If I were able to kill it, it would instantly cut off more than half of the spam. But, it’s the first permanent e-mail address I ever got, using the excellent pobox.com mail forwarding service. I’ve had it for almost 15 years. Because it’s so old, I’m extremely reluctant to part with it — what if that’s the only address a long lost friend has?

Well, it looks like I can have my cake and eat it too. pobox.com just introduced new spam filtering controls and services, which are far more effective than the old filters that were enabled on my account. Last night I turned them on, and already the amount of spam coming into my pobox.com e-mail address has dropped to almost zero.

I wouldn’t exactly call this the turn of the tide, but it’s certainly encouraging. Because it’s my only hope to avoid having to look at 100,000 spam messages in 2004, which is where the growth curve points, if there isn’t change.

I’ll let you know how it’s looking when Q2 is over.

The idea is, if my e-mail address is “michael a-t alderete.com”, then there are only a few servers on the internet that are likely, or permitted, to send e-mail for the alderete.com domain. When you receive an e-mail from that address or domain, if you knew which servers on the internet were legitimate senders, then you could reject messages from all other servers.

This is useful because it’s common practice by spammers to forge the From: header of their spam messages, and because they are almost never able to send those messages from the real server for the domain. (This is why bouncing spam back to the sender just makes the spam problem worse.)

I had incentive to do this because one of my e-mail address domains, alderete.com, has been forged heavily recently (though not quite “Joe Job“ed), with thousands of e-mails being sent out with forged from addresses like “Tammeravxryawwv@alderete.com” and “Glenniedatjklcjyknai@alderete.com”. When the spams bounce back, they come to my Inbox. Thousands of them.

Now, SPF isn’t a panacea for this problem, mostly because there has not been a lot of deployment of the technology yet. But that’s coming; AOL recently began trialing it, and if it’s successful I am sure the other big ISPs will do so soon.

When they do, I’ll be ready to reap the benefits.

QUALCOMM’s Eudora has been my e-mail client of choice for nearly 10 years, and last week a major new version shipped, Eudora 6. I’m usually of the “fools rush in” school of thought with regards to software updates, so I waited to see what people were saying about the upgrade (MacInTouch is a great resource for these “reader reports”).

But it’s been a week, and nary a peep. And with the amount of spam I receive continuing to grow, I really wanted to try the new SpamWatch feature. So, after doing multiple backups, I upgraded myself over the weekend.

My primary concern was whether and how my other anti-spam tool, Spamnix, would work with the new version, especially with the new SpamWatch feature. Unlike a lot of other third-party anti-spam tools, Spamnix is a Eudora plug-in, and so runs “in-process” (i.e., inside) with Eudora. [Update: SpamSieve 2 just shipped, and now also includes a Eudora plug-in. Very cool!] This makes it more efficient, but also (in theory) more susceptible to compatibility issues.

I’m happy, nay, thrilled to report that Spamnix works fine with Eudora 6 (for Mac OS X), and that Spamnix + SpamWatch is more effective than either tool alone.

I love the way that SpamWatch and Spamnix tag-team to combat spam. SpamWatch gets first crack, before other filters or plug-ins look at the message, and if the message’s score is over the spam threshold, it will be filtered into the Junk mailbox, with no further processing. (Qualcomm designed SpamWatch to run first, and you can’t change that.)

If a message doesn’t get caught by SpamWatch, then Spamnix takes a look at it, and if Spamnix decides it’s spam, it’ll go into Spamnix’s own spam folder (on my system named “ Spamnix”; note the initial space to influence sort order). These messages, nicely separated and usually all spam, are prime candidates for further training for SpamWatch.

I receive hundreds of spam messages a day, but after two tiers of spam filtering very little spam gets to my Inbox — so far only a couple a day, with very little training of SpamWatch yet. The few that have made it through have gone straight back to SpamWatch for training. :-)

What is fascinating about this process is the progress that SpamWatch has made, in less than 4 days of processing my mail. The first time I downloaded a sizeable batch of e-mail (more than 50 messages), most of the spam got through SpamWatch, and caught by Spamnix. After training SpamWatch with those messages, and then downloading another big batch a few hours later, the ratio went the other way: SpamWatch was now catching most spam before Spamnix got a chance to look at it.

I’m still glad to have both layers. Spamnix was extremely effective at catching my spam, prior to SpamWatch being added to the mix, and it’s still catching spam that SpamWatch is missing. So overall, I am doing better in my personal war against spam (though it’s important to remember that this is defensive action only).

About the only downside of introducing SpamWatch as a new layer of anti-spam defense is that right now it’s relatively untrained, and generating a larger number of false positives (non-spams filed in the Junk folder) than I’m used to. SpamWatch ships “pre-trained”, meaning it already has a database of spam words to run against, but this list is generic, not customized to my own e-mail traffic. So it’s not that surprising that some of Rochelle’s e-mails are getting tagged as spam. My previous experience with Bayesian filtering is that it rapidly adjusts as you correct its mistakes, so I’m confident the false positives will go down in a week or so.

At any rate, I’m quite happy with the new version, especially since I was still in my 12 month support period from my last upgrade, so version 6 was free. Recommended, even if you have to pay for it.

His motion to dismiss the case was so effective that the plaintiffs are now trying to back out of the case, so they can avoid having to pay opponent’s legal fees, which they’re likely to have to do. Pete’s not going to let them do that.

Wellborn has been so effective at racking up successes against spammers, to the tune of multi-million dollar judgments, that he’s called the “Spammer Hammer.” And after defending on this lawsuit, he’s switching to offense, to run down the toads behind it for their spamming activities.

Go get ‘em, Hammer!

I do still get some spam, though. Fortunately, Jon Udell’s recent weblog entries and review at InfoWorld turned me onto a replacement that is free, and will remain so (it’s Open Source): SpamBayes.

Like SpamNet, it can be installed as an Outlook plug-in, and easily used via buttons on Outlook’s toolbar. But the technology behind it is very different, as it uses Bayesian filtering rather than distributed recognition. It’s also different in that the core project and recognition engine is command line-oriented; the Outlook-only plug-in is terrific, but only a side project. It’s not required, and there are plenty of ways for those who use something other than Outlook for e-mail to use SpamBayes.

You can read the review for a thorough look, but my experience was that it was just as easy to install as SpamNet, is extremely effective at blocking spam, and is also having fewer false positives. I think the reason for that is SpamNet uses other people’s spam reports to decide what to block in my Inbox, and there’s a lot of people who just block e-mails they signed up for (newsletters, promos, etc.), rather than unsubscribe from them. Those false reports pollute the knowledge base, and affect my results. Bayesian filtering is exactly the opposite — it only cares what I think is spam.