When you get more than one thousand spams a week, you live for improvements of a couple of tenths of a percent. I of course upgraded immediately.

It was a little anticlimactic. Given the minor delta between my 99.5% accuracy and John’s 99.9%, I did not see an immediate difference in my spam protection. It remains very, very good. (Surprise. Given the small difference in accuracy, it will take months to have something to compare.)

Tonight I got around to reading the more detailed version history, which explained the improvements in a bit more detail:

Made lots of improvements to SpamSieve’s parsers and tokenizer for better accuracy. To fully take advantage of this, you will need to reset SpamSieve’s corpus and re-train it (e.g. with 300 recent good messages and 600 recent spams). However, this is certainly not required, and I expect that most people will opt for the simpler upgrade of just installing the new SpamSieve application. [Emphasis added.]

Given the amount of spam I get, I want to achieve the maximum level of accuracy, not opt for the easiest upgrade path. So, curious, I checked my Junk mailbox in Eudora: 598 spam messages. A sign from god. I collected 300 recent good messages, backed up my SpamSieve corpus, and reset it. Just finished retraining.

We’ll see how it goes. In a month or two.

This is all the more remarkable given their tagline (“Reclaim Your Inbox”), and the second sentence of their Why Use Thunderbird blurb: “We designed Thunderbird to prevent viruses and to stop junk mail so you can get back to reading your mail.” Thunderbird is positioned as the more user-centric, safer alternative to Microsoft Outlook.

For the most part, that’s well-deserved. Thunderbird isn’t riddled with security problems like Outlook, and it comes with built-in anti-spam features that are quite a bit more effective than those that are built into Outlook.

But the reality is that, even if your email client isn’t itself a security disaster waiting to happen, if you use Windows, you need anti-virus protection, because you’re definitely going to get viruses and worms and trojans in your Inbox, and once they’re in your Inbox, one wrong click and your computer is hosed. And it’s in dealing with these threats where Thunderbird falls down.

In the mozilla.org Thunderbird Help section, this is the only “useful” information about what to do about viruses:

As with any mail program, take proper caution before running any file that you receive in e-mail. Appropriate anti-virus software should also help keep you safer.

With a little more work, you might follow links to a third-party site with an anti-virus knowledgebase article that is by turns unusable, out of date, and a recommendation against Thunderbird. You definitely finish reading that article thinking that no major anti-virus software vendor supports Thunderbird — and you’d be right.

In Rochelle’s case, Norton Anti-Virus has corrupted her Thunderbird Inbox multiple times, causing her to lose all of her messages. (Fortunately she’s mostly using Gmail these days.) I’ve spent hours looking into this, and there is basically no good configuration for using Norton and Thunderbird together. (It does not help that Norton has a truly abysmal user interface.) I’d switch Rochelle to McAffee, except I can’t find good configuration information for that combination, either.

The worst part is the Thunderbird developers seem to take the attitude that the problem is the anti-virus software vendors’ fault. This defect was reported in 2001, and is still marked as open in their defect tracking system. After four years of end users losing email, I would think they might start to realize that they need to work with the commercial anti-virus software vendors to get compatibility.

Failing that, there are some pretty good Open Source anti-virus tools for Windows. How about incorporating one of those into Thunderbird? Or picking one to make the “official” anti-virus software to use with Thunderbird, and giving clear, complete directions for how to install and use it with Thunderbird? (I’ve read good things about ClamWin.)

I can understand the impulse to blame the anti-virus vendors for not working with Thunderbird. Technically, it is their fault. But from the perspective of a Thunderbird user, rather than developer, I just want Rochelle’s Inbox to be protected, both from incoming malware and from the scanners that do the protecting.

Right now, Thunderbird the product can’t provide that. From an end-user’s perspective, that’s Thunderbird’s fault.

So I did a few things about it.

WordPress 1.5 has a number of new features that make dealing with comment spam a little easier, and I’ve adopted those, while also disabling all of the anti-spam plug-ins I had been using in WordPress 1.2. I also recently implemented some suggestions for making it a little harder for automated commenting, adopting a few of the ideas from “A short monograph on the theme of blog comment spam”:http://www.simong.org/index.php?p=739, and a couple from “Tom Raftery”:http://www.tomrafteryit.net/comment-spam-plugins-no-longer-required/.

I had a problem with the fake form I implemented, because of a misplaced “%”, which caused it to show up and screw up the regular form. That’s fixed, and I think the fake form should be invisible to everyone using a modern browser.

Hard to say how effective it really is at this point, too soon to tell. But at least so far, no spam comments have been left since I implemented the changes and new traps.

But in my own, more personal battles with spam I’ve been more successful at holding back the tide. My stats for 2004:

Filtered Mail
36278 Good Messages
72239 Spam Messages (67%)
197 Spam Messages Per Day

SpamSieve Accuracy
135 False Positives
451 False Negatives (77%)
99.5% Correct

Nearly seventy five thousand spam messages came at me, but thanks to SpamSieve a mere 451 made it into my Inbox. That’s less than two spams a day. Simply amazing.

Even more amazing is that the second half of 2004 was far better than the average for the entire year, meaning that while the rest of the internet went to hell, my own situation actually improved as the year went on:

Filtered Mail
15729 Good Messages
32650 Spam Messages (67%)
176 Spam Messages Per Day

SpamSieve Accuracy
53 False Positives
181 False Negatives (77%)
99.5% Correct

I would attribute this improvement primarily to the new filters I added to my oldest, most spammed email address, and the deactivation of a few temporary addresses I created.

Interestingly and unfortunately, the spam volume seems to have begun to go up again in 2005, as so far I’ve averaged 199 spam messages a day this year. I’ve expired a few more highly targeted addresses, though, in hopes of beating some of it back. Time will tell…

First of all, I’m running WordPress 1.2.2 (and all of its prerequisites, e.g., Apache, MySQL, PHP, etc.). After installing WordPress, the next thing I did was install the Kubrick template, which replaces the default index.php, wp-comments.php, and wp-layout.css files with new versions that use a much more attractive layout, design, etc.

Then I tweaked the fonts in the CSS file (I’m using Trebuchet for the body copy, and Georgia for the headers), moved a bunch of things around (like the syndication feed links), and deleted other things (like the monthly archive links) inside the index.php file. And I encoded my e-mail address with the Automatic Labs Enkoder, to protect it from spam harvest bots.

So far, none of this is rocket science, or hard to figure out what I did. A few of the less obvious things.

I have the Textile 1 plug-in, which is part of the WordPress 1.2 distribution, turned on (it defaults to off), which converts straight quotation marks and apostrophes to curly versions, double-dashes to “em”-dashes, etc., to make the text a little prettier. This improved typography and ease of formatting is one of the major reasons I switched to WordPress.

I changed the default format for permalinks (configured on the Options » Permalinks screen of the administrative console), to duplicate the link structure used by my old weblog software. Then I manually added a dozen additional rewrite rules to ones generated by WordPress, to handle all the different kinds of permalinks people were using with the old software. The details of all this will get its own post eventually.

For articles where I show only an excerpt on the front page, I am displaying a word count in the “Read the rest…” link. This is done with the Words In Post plug-in (which I hacked a little bit so it would work in the the_content() call in index.php).

On the permalinks pages, I use Binary Bonsai’s plug-in for Dunstan’s Time Since to do the relative dates (“posted X days, Y minutes ago”).

I am using the Related Posts plug-in to display links to posts which may possibly be related to the current post. I initially wasn’t happy with the related items it was finding, but I modified the plug-in’s code to slightly change the way the it uses keywords, and it seems to be doing a bit better at finding posts that are genuinely related.

For a really trivial tweak, I’m using the Nice Categories plug-in, which basically just adds an “and “ to the list of categories a posting is filed under in the metadata section at the end of the individual post page. (This is a perfect example of how I can spend a couple of hours dicking around on this site, with little visibly accomplished.)

I was using the Staticize Reloaded plug-in to do automatic caching of the web pages, but that caused problems with the relative date, and I haven’t converted that to use a JavaScript version of Time Since yet, so I’ve got Staticize turned off for the moment. But it’ll go back on in the future.

To control comment and trackback spam, in addition to filling WordPress’s built-in comment moderation watch list with almost 2000 keyword and IP addresses, I’ve got two plug-ins enabled, Kitten’s Spaminator and ThreeStrikesSPAM. So far, no comment or trackback spam has made it through. Knock wood.

That’s it for now, but I’m sure there will be more later…

It turned out the issue was a couple of the PHP “open” tags — the text tokens that are used to signal that what follows is PHP code, not HTML code. The Kubrick template (at least as of version 1.2.6) uses the standard “full” tags almost everywhere, but there were a few instances where the developer used the short versions. Since I have short tags disabled on my server, it was causing a parse error, the end result of which was that the comments form could not be displayed. Fixed.

Now that I’ve got comments working, I switched to allowing comments on postings by default, and letting them post without moderation first. We’ll see how that goes.

Because, the weblog world is dealing with a plague of comment spam these days, from “search engine optimization” scumbags who are trying to promote their (or their clients’) websites. So I’ve also installed and enabled a number of comment spam plug-ins. We’ll see how that goes, too, though I’m somewhat pessimistic, for a slew of reasons. Probably I’ll have to turn moderation back on at some point…

And I installed a new caching plug-in, which should theoretically improve the performance of this site. Not really a problem with the volume of visitors I’m getting today, but you never know, and having the plug-in active will hopefully prevent unpleasant surprises.

My very first anti-spam tool was something called Mailfilter. I used it for my personal e-mail on Mac OS X, wrote about it here, and almost immediately afterwards lost a non-spam message to an aggressive keyword match. That was the end of Mailfilter. I can’t even remotely recommend it, as it’s just not intelligent enough (strict, single expression matching), and had zero safety net.

My next attempt at a solution was a utility called SpamFire. Like Mailfilter, it is a “pre-filter,” which means it would run before my e-mail client, download my mail, and skim out the spam. Unlike Mailfilter, it actually saved the trapped messages, so if it made a mistake, I could recover the message. It had plenty of other differences from Mailfilter, which I wrote about previously, and which made it so useful that it became the first anti-spam tool I paid for. But in the end I switched to a different tool because SpamFire was separate from my e-mail client, and that made it cumbersome to use.

In the meantime, I had spam coming into my e-mail at work, and at the recommendation of a co-worker, I installed Cloudmark’s SpamNet (now called SafetyBar), an add-in for Microsoft Outlook. It worked reasonably well, but then they went and started charging money for it. It didn’t work well enough, and I didn’t get enough spam at my work address, for it to be worth paying for, so I stopped using it.

I replaced it with another plug-in for Outlook called SpamBayes. Even more effective than SpamNet, it confirmed for me the value of having a tool that plugs into and works directly with your e-mail client. It made dealing with spam seamless, almost effortless (given the lower volume of spam than my home e-mail addresses). Unlike SpamNet, it’s Open Source, and also unlike SpamNet, it doesn’t depend on a third-party server to run. If I still used Outlook (or had a “work” e-mail address), I would still be using SpamBayes. Effective, easy to use, and free. By far the best anti-spam solution I’ve found for Outlook on Windows. Highly recommended.

My happiness with SpamBayes for Outlook lead me to search for an anti-spam tool for my personal use which integrated fully with my e-mail client, Eudora for Mac OS X. I found Spamnix, which wrapped the Open Source tool SpamAssassin in a Eudora plug-in. I fell instantly in love and dropped SpamFire. This was the second anti-spam tool I paid for.

Spamnix was great for a number of reasons. Its interface within Eudora took the form of a new mailbox, and a couple of simple menu commands. Incoming mail judged likely to be spam was shunted off to the new mailbox; the menu commands let you rescue items that were not spam, etc. It was simplicity itself to use, and fairly effective at trapping spam.

The failing of Spamnix was that it was based on an earlier version of SpamAssassin, which while it had some great regular expressions and other traps for catching spam, it didn’t include any Bayesian filtering at all. This meant that spam filtering was good, but never improved. However, it took a new version of Eudora for me to realize that this mattered. (Spamnix has since been upgraded substantially, and uses the latest SpamAssassin with Bayesian filtering; but I’ve not tried it, I moved on to other things, as you can continue to read.)

SpamWatch was a plug-in shipped with Eudora 6, which I paid for, making it the third anti-spam tool for which I shelled out money. It provided me with Bayesian filtering in my Mac OS X mail environment for the first time. It was a revelation. The first time I downloaded mail after upgrading, SpamWatch caught a half-dozen spam messages, and Spamnix caught several dozen. A quick menu choice fed the missed spam to SpamWatch, and forever afterwards the ratio was reversed. I’ll never again think the same way about learning systems; Bayesian filtering really becomes dramatically more effective the more you use it. I had learned this with SpamBayes at my work e-mail, but I simply didn’t get enough spam at that address for it to really shine.

SpamWatch was so effective that it soon spelled the end for Spamnix. I just didn’t need a second spam filter (and the attendant delay in getting my mail while it processes incoming messages) when the first filter was so good. So I uninstalled Spamnix, and life was good.

SpamWatch actually improved my effectiveness in fighting spam, not just by being a better filter. The new version of Eudora added a new field for mail messages, the Junk Probability, a 1-100 score on how likely a message was judged to be spam. I quickly learned to process my Junk mailbox by first sorting by the Junk Probability, and then scanning for false positives. By sorting first, the messages at the beginning of the list were a lot more in need of review than messages at the end, making it possible to skim more rapidly over items that were certainly spam.

I don’t really remember why I decided to give SpamSieve a try. I do remember noticing that SpamWatch seemed to have hit a plateau, and wasn’t improving any more. I had friends who used and liked SpamSieve, and were getting better stats than I was with SpamWatch, but the interface between SpamSieve and Eudora was, for a long time, through AppleScript, and the integration wasn’t smooth enough. Probably when the version that used a Eudora plug-in came out, I decided what the heck, it won’t cost anything to try it a couple days, and see how it does.

It was not the revelation that SpamWatch was, but it only took a training pass at my archived spam and Inbox (a few thousand messages between them), and a couple of days use proved that it was noticeably better than SpamWatch. After a week I was sold, and opened my wallet, a fourth time, taking my anti-spam expenditures over $100. (Compared to the problem I have with spam, that’s not a lot of dough, money well spent. But it chafes when you realize that the spam problem is entirely due to greedy amoral scumbags who have polluted the e-mail highways and byways to make a few cents per pound of pollutant spread.) I also disabled SpamWatch, since SpamSieve entirely replaced it.

Well, not entirely, not at first. Although SpamSieve had a more sophisticated and accurate Bayesian engine at its core, there was one thing that it didn’t do well, which was assign a Junk Probability to each message. This meant that I lost a very effective tool in my Junk folder processing, of being able to sort the caught messages from least to most likely spam.

I e-mailed Michael Tsai, the developer of SpamSieve, explaining how I used that feature of SpamWatch. He was highly responsive, and agreed that my approach sounded genuinely useful. Unfortunately, at that time SpamSieve’s engine’s algorithms and formulas didn’t really generate a score that could be usefully assigned to Eudora’s Junk Probability column. The math just didn’t work that way. He said he’d look into it, and perhaps in the future, perhaps it might do this.

Fortunately for me, the future did eventually come to pass, and SpamSieve now does assign useful probability scores to processed mail. Its final fault gone, and with the best scoring engine and perfect integration with Eudora, it is by far the best tool I’ve used for managing the amount of spam I get, and I’m happily using it today. Compatible with virtually every e-mail client running on Mac OS X, it totally deserves its reputation as being the best anti-spam tool available on my platform of choice. Highly, highly recommended.

After all that, you’d think I’d be wrapping this posting up. You and I both wish that was true. But no. I’ve got five more tools to write about.

First is Thunderbird, the stand-alone e-mail client that evolved out of the Mozilla suite. It’s available for Mac OS X, Windows, and Linux, and other platforms, too, I think. It’s a nice enough e-mail client, and one of the best for sending and receiving HTML e-mail, if you’re the sort of person who likes that kind of thing.

It has gotten quite a bit better since I evaluated it, to the point where I installed it on Rochelle’s PC, as a replacement for Netscape Communicator, which she had been using since the mid-‘90s. It has a built-in Bayesian spam filter, which works well enough. It’s definitely not as accurate as other Bayesian classifiers I’ve used; SpamBayes and SpamSieve are both quite a bit better, and I think Eudora’s SpamWatch might be a little better. But it’s more than good enough for Rochelle, who gets an order of magnitude fewer spam messages than I do.

Another advantage of Thunderbird is that you’re not using Outlook, which is the number one attack vector for viruses and worms. It’s a good tool, and if you’re stuck on Windows and can’t do better, it’s definitely worth switching to. Recommended.

Another e-mail client I haven’t used much is the bundled OS X e-mail client, Mail.app. It doesn’t work the way I do, and doesn’t seem to be well-suited to managing the volume of mail I get on a daily basis. Supposedly it has good junk mail controls (which use latent semantic analysis), but in the testing I did with it on a secondary e-mail account, it didn’t seem that good, somewhere around the Spamnix level of accuracy. That is, very good, but not excellent. If you like Mail.app, since you can use SpamSieve with it, that’s what I’d recommend doing, and blow off the built-in junk mail controls.

Finally I come to the server-based tools. I’ve saved these for last, not because I tried them last, but because most people won’t be able to use them themselves. Unless you run your own mail server, most of these are impossible to use.

The first server-side solution I tried, and used for quite some time, was what are known as RBL lists. RBL stands for “Real-time Blackhole List”, and the way it works is as mail is received, the mail server sending the message to your server is looked up on a list, which contains known-bad e-mail servers (ones that are known to send spam). If the foreign server is on the list, the e-mail is rejected, in real-time.

In theory this should be extremely effective, and indeed it does cut down on spam considerably. But not completely, and not without collateral damage. There are just too many servers out there that are sending spam, most of them home PCs that have been infected with a worm or virus that converted it to a “zombie” that sends out millions of spam e-mails. It’s impossible to keep the lists current.

It’s also impossible to keep them accurate. There’s no way to maintain the lists with perfect accuracy, mistakes are inevitable, due to both ignorance and malice on the part of people submitting candidates. In the end I had too many messages rejected that were OK, and I had to turn the RBLs off. (I may turn them on again, I go back and forth on the “damn the consequences” philosophy…)

I also tried MIMEDefang, which was a wrapper for SpamAssassin and Vipol’s Razor, with a plug-in for my mail server, Sendmail. I ended up losing mail to this solution, when it would generate errors under load. The failure was intermitent, and impossible to reproduce on demand. Since the mail would just get dropped on the floor, completely lost, I decided I couldn’t afford to try to track down the issue, and simply uninstalled all of it. I’ll surely try another wrapper for SpamAssassin at some point, when I’ve converted my mail server to Postfix, for which it is easier to write plug-ins, and should therefore be easier to write reliable plug-ins.

The last server-side tool I’ve been using doesn’t live on my server. Maybe that’s why it’s worked so well. I wrote about the amount of spam that was coming to my oldest e-mail address at pobox.com, and how the new filters they rolled out saved the address from deletion. I’m still really happy with the results of that service, and plan to keep my pobox.com address for the foreseeable future. Recommended.

Filtered Mail

6674 Good Messages
16976 Spam Messages (72%)

SpamSieve Accuracy

52 False Positives
71 False Negatives (58%)
99.5% Correct

And here are the stats for the first half of the year:

Filtered Mail

20554 Good Messages
39595 Spam Messages (66%)

SpamSieve Accuracy

83 False Positives
270 False Negatives (76%)
99.4% Correct

You can review my previous post for the Q1 details, but the overall trends are:

• SpamSieve increased its overall accuracy, but the number of false positives (legit messages marked as spam) went up, which is a bit troubling.
• The actual amount of spam I received was down, but as a percentage of my mail stream, it was up. (This is because my legitimate mail traffic is down 50% after I unsubscribed from some mailing lists.)
• 75% less spam reached my Inbox (71, vs 270 in Q1).

The overall reduction in spam traffic is encouraging, and proof that the pobox.com mail filters that I enabled at the end of Q1 are doing a tremendous job at rejecting spam coming to my oldest e-mail address.

The one really irritating new trend this past quarter, which doesn’t show up in the stats, is the number of bounces and anti-virus error messages hitting me. The alderete.com and aldosoft.com domains have been spoofed (forged) in a lot of spam and worm traffic, and because I have my domains “wild carded” to accept e-mail for “any-address@alderete.com”, the result is I get a lot of crap in my Inbox that is intended to let “pigxwnslesps@aldosoft.com” know that the crap didn’t go through.

So I’ve been compiling a list of the e-mail aliases I actually use (quite a lot because I use custom addresses with various vendors and web sites, so I can track who sells my address to spammers, turn off specific addresses, etc.), with the intent that I’ll enable those, and disable the wild cards. I hope to do that over the holiday weekend, and expect it to have almost as dramatic effect on crap coming in as the improved pobox.com spam filters.

In spite of these numbers, I have two things that give me hope.

First, SpamSieve is an amazing anti-spam filter that integrates well with Eudora. It’s far more reliable than the built-in SpamWatch feature that debuted in Eudora 6, primarily in the area of false positives (real messages mistakenly filtered out):

Filtered Mail

13565 Good Messages
22255 Spam Messages (62%)

SpamSieve Accuracy

21 False Positives
197 False Negatives (90%)
99.4% Correct

SpamSieve is award-winning software for Mac OS X, and it integrates beautifully with both Eudora and Mailsmith, the two best e-mail clients for the platform. I am getting to the point where I trust SpamSieve enough to just purge filtered e-mail without reviewing it.

Without SpamSieve, I would be going insane because of spam.

The second thing I have on my side is that more than half of my spam comes to one e-mail address, the oldest e-mail address I still use. If I were able to kill it, it would instantly cut off more than half of the spam. But, it’s the first permanent e-mail address I ever got, using the excellent pobox.com mail forwarding service. I’ve had it for almost 15 years. Because it’s so old, I’m extremely reluctant to part with it — what if that’s the only address a long lost friend has?

Well, it looks like I can have my cake and eat it too. pobox.com just introduced new spam filtering controls and services, which are far more effective than the old filters that were enabled on my account. Last night I turned them on, and already the amount of spam coming into my pobox.com e-mail address has dropped to almost zero.

I wouldn’t exactly call this the turn of the tide, but it’s certainly encouraging. Because it’s my only hope to avoid having to look at 100,000 spam messages in 2004, which is where the growth curve points, if there isn’t change.

I’ll let you know how it’s looking when Q2 is over.

The idea is, if my e-mail address is “michael a-t alderete.com”, then there are only a few servers on the internet that are likely, or permitted, to send e-mail for the alderete.com domain. When you receive an e-mail from that address or domain, if you knew which servers on the internet were legitimate senders, then you could reject messages from all other servers.

This is useful because it’s common practice by spammers to forge the From: header of their spam messages, and because they are almost never able to send those messages from the real server for the domain. (This is why bouncing spam back to the sender just makes the spam problem worse.)

I had incentive to do this because one of my e-mail address domains, alderete.com, has been forged heavily recently (though not quite “Joe Job“ed), with thousands of e-mails being sent out with forged from addresses like “Tammeravxryawwv@alderete.com” and “Glenniedatjklcjyknai@alderete.com”. When the spams bounce back, they come to my Inbox. Thousands of them.

Now, SPF isn’t a panacea for this problem, mostly because there has not been a lot of deployment of the technology yet. But that’s coming; AOL recently began trialing it, and if it’s successful I am sure the other big ISPs will do so soon.

When they do, I’ll be ready to reap the benefits.

QUALCOMM’s Eudora has been my e-mail client of choice for nearly 10 years, and last week a major new version shipped, Eudora 6. I’m usually of the “fools rush in” school of thought with regards to software updates, so I waited to see what people were saying about the upgrade (MacInTouch is a great resource for these “reader reports”).

But it’s been a week, and nary a peep. And with the amount of spam I receive continuing to grow, I really wanted to try the new SpamWatch feature. So, after doing multiple backups, I upgraded myself over the weekend.

My primary concern was whether and how my other anti-spam tool, Spamnix, would work with the new version, especially with the new SpamWatch feature. Unlike a lot of other third-party anti-spam tools, Spamnix is a Eudora plug-in, and so runs “in-process” (i.e., inside) with Eudora. [Update: SpamSieve 2 just shipped, and now also includes a Eudora plug-in. Very cool!] This makes it more efficient, but also (in theory) more susceptible to compatibility issues.

I’m happy, nay, thrilled to report that Spamnix works fine with Eudora 6 (for Mac OS X), and that Spamnix + SpamWatch is more effective than either tool alone.

I love the way that SpamWatch and Spamnix tag-team to combat spam. SpamWatch gets first crack, before other filters or plug-ins look at the message, and if the message’s score is over the spam threshold, it will be filtered into the Junk mailbox, with no further processing. (Qualcomm designed SpamWatch to run first, and you can’t change that.)

If a message doesn’t get caught by SpamWatch, then Spamnix takes a look at it, and if Spamnix decides it’s spam, it’ll go into Spamnix’s own spam folder (on my system named “ Spamnix”; note the initial space to influence sort order). These messages, nicely separated and usually all spam, are prime candidates for further training for SpamWatch.

I receive hundreds of spam messages a day, but after two tiers of spam filtering very little spam gets to my Inbox — so far only a couple a day, with very little training of SpamWatch yet. The few that have made it through have gone straight back to SpamWatch for training. :-)

What is fascinating about this process is the progress that SpamWatch has made, in less than 4 days of processing my mail. The first time I downloaded a sizeable batch of e-mail (more than 50 messages), most of the spam got through SpamWatch, and caught by Spamnix. After training SpamWatch with those messages, and then downloading another big batch a few hours later, the ratio went the other way: SpamWatch was now catching most spam before Spamnix got a chance to look at it.

I’m still glad to have both layers. Spamnix was extremely effective at catching my spam, prior to SpamWatch being added to the mix, and it’s still catching spam that SpamWatch is missing. So overall, I am doing better in my personal war against spam (though it’s important to remember that this is defensive action only).

About the only downside of introducing SpamWatch as a new layer of anti-spam defense is that right now it’s relatively untrained, and generating a larger number of false positives (non-spams filed in the Junk folder) than I’m used to. SpamWatch ships “pre-trained”, meaning it already has a database of spam words to run against, but this list is generic, not customized to my own e-mail traffic. So it’s not that surprising that some of Rochelle’s e-mails are getting tagged as spam. My previous experience with Bayesian filtering is that it rapidly adjusts as you correct its mistakes, so I’m confident the false positives will go down in a week or so.

At any rate, I’m quite happy with the new version, especially since I was still in my 12 month support period from my last upgrade, so version 6 was free. Recommended, even if you have to pay for it.

His motion to dismiss the case was so effective that the plaintiffs are now trying to back out of the case, so they can avoid having to pay opponent’s legal fees, which they’re likely to have to do. Pete’s not going to let them do that.

Wellborn has been so effective at racking up successes against spammers, to the tune of multi-million dollar judgments, that he’s called the “Spammer Hammer.” And after defending on this lawsuit, he’s switching to offense, to run down the toads behind it for their spamming activities.

Go get ‘em, Hammer!

Problem is, I haven’t found a better e-mail client than Netscape. Outlook and Outlook Express are out of the question. They are deeply insecure applications, and the number one vector for spreading computer viruses. (Mark my words, in the next 12 months there will be a malignant virus that will wipe Outlook users’ hard disks clean. It’s just a matter of time.) They are also spam-friendly applications (though an Open Source project, SpamBayes, gives Outlook robust anti-spam tools). People who voluntarily use Outlook or Outlook Express are stupid. IM!HO.

I actually bought Eudora Pro for Windows for Rochelle’s computer, on the basis of my experience using Eudora on the Mac for the last decade. But Eudora for Windows uses the obsolete Windows MDI interface paradigm, where all of the windows are contained in one “parent” window. It’s maddening, and a relic from the late 80s. The application has a number of other quirks, differences from the Mac version, to the point where I found it unusable.

So I’m evaluating Thunderbird, to see if it’s ready for Rochelle. I plan to use it regularly over the next few weeks, configured to manage one of my less-used e-mail accounts.

It’s a good thing it’s a less-used e-mail account, because already in my first 15 minutes, it’s clear that Thunderbird is still pretty raw (giving double meaning to the “trying” in this post’s title). Basic e-mail functionality is there, and the application seems solid (no crashing); this is the result of Thunderbird’s gestation as part of the Mozilla Suite. You can use, and even rely on Thunderbird. But there are a lot of fit-and-finish issues, which seem like small things, but add up to making it unsuitable — unenjoyable — for daily use.

Some examples:

1. The first thing I want to do when setting up an e-mail client is turn off automatic downloading of HTML images. (Loading images in a spam message can tell the spammer your e-mail address is valid, resulting in a lot more spam.) There is a control for this in Thunderbird’s preferences, hidden a little too deeply (Advanced -> Privacy -> Block loading of remote images), but easily checked once you find it. So far so good.

The problem is when you get messages with graphics from valid senders. The graphics don’t display, as per the general preference, but there’s no way to override that for the one valid message. This renders some messages unreadable.

Solution: a toolbar button in the message window to download that message’s graphics.

2. The default font settings render many messages a blur, with the text far too small to be legible. (This is on Mac OS X, it might be better on Windows or Linux.) The “minimum size” preference seems to do nothing, and the View -> Text Zoom menu option does not appear to be a global setting. I finally solved the issue by changing my Serif font setting to Lucida Grande, a sans serif font that is highly readable, even at small sizes. But all in all, there are far too many settings and options that affect text size and font choice, and it’s not at all clear what does what, how they interact, or how to accomplish specific goals with regard to text rendering.

It reminds me of Don Norman’s description of refrigerator / freezer settings in a href=“http://www.amazon.com/exec/obidos/ASIN/0465067107/haightlifecom”>The Design of Everyday Things. In most home refrigerators the freezer and refrigerator compartments share a single compressor, the key component of the cooling system. Because it’s shared, making changes to the freezer setting, e.g., setting it lower, can affect the refrigerator setting, making it lower too. So you turn up the refrigerator knob to keep your lettuce from freezing, but that makes the freezer less cold, and your ice cream oozes out of the carton. You have to fiddle and fuss to finally get to a balance you can live with.

It’s a ridiculous thing for an end-user to have to deal with, and it happens because the designers give you controls that affect the system’s internals directly, instead of letting you choose a goal state (e.g., a specific temperature for each compartment), and have the system figure out how to achieve it. Product designers and programmers do this because it’s easy to build, and because they don’t see anything wrong with it. The problem is that users don’t think like programmers, and have trouble figuring things out.

Thunderbird is supposed to be a simplified, easy-to-figure-out e-mail client, vs. the everything-but-the-kitchen-sink e-mail client in the Mozilla Suite. They have a ways to go with the text settings.

3. Thunderbird makes some assumptions about my e-mail reading workflow that are wrong. If I open a message, read it, and then delete it, Thunderbird automatically opens the next message, in a window sized and positioned exactly like the first message.

First of all, while this straight-through workflow may work for some people, it’s deeply distracting to me. I pick and choose my e-mails, working via priority order (or whim), not on the order the messages arrived. I suspect most sophisticated e-mail users do this. Auto-opening a message I would prefer to defer looking at just means I have to close it, and then right-click to mark the message as Unread. Pain in the ass. There appears to be no way to affect this behavior.

The second issue with this is that Thunderbird’s screen redraws are extremely efficient. There is zero flicker when one message disappears and the other appears. Because the new message appears in the same place and is the same size, only the text changes. If you’re looking at a new message that is visually similar to the previous one — say, two text messages — you might not notice it was new, and think that you didn’t hit delete at all. Guess what you’d do then.

These are three examples, but I’ve seen many other issues. I can hold my nose and manage this low-priority e-mail account, but it’s clear that Thunderbird has a few more months of development in front of it before I’ll give it to Rochelle.

I do still get some spam, though. Fortunately, Jon Udell’s recent weblog entries and review at InfoWorld turned me onto a replacement that is free, and will remain so (it’s Open Source): SpamBayes.

Like SpamNet, it can be installed as an Outlook plug-in, and easily used via buttons on Outlook’s toolbar. But the technology behind it is very different, as it uses Bayesian filtering rather than distributed recognition. It’s also different in that the core project and recognition engine is command line-oriented; the Outlook-only plug-in is terrific, but only a side project. It’s not required, and there are plenty of ways for those who use something other than Outlook for e-mail to use SpamBayes.

You can read the review for a thorough look, but my experience was that it was just as easy to install as SpamNet, is extremely effective at blocking spam, and is also having fewer false positives. I think the reason for that is SpamNet uses other people’s spam reports to decide what to block in my Inbox, and there’s a lot of people who just block e-mails they signed up for (newsletters, promos, etc.), rather than unsubscribe from them. Those false reports pollute the knowledge base, and affect my results. Bayesian filtering is exactly the opposite — it only cares what I think is spam.

Seriously, this is clearly an ambitious man with his finger squarely on the pulse of what’s making people feel crazy. He takes the pulse, he prosecutes cases against the bad buys. It’s great. I wish I had a politician so responsive in California. I predict that state attorney general is not the highest office he will ever hold.

Now, can I sign up to be on the jury?