The Different Categories Of Junk Email Filters
Since spam has become such a large irritation and source of lost time,
(with no immediate resolution in sight) more and more companies are
looking for solutions against this headache, not only for the employee
time saved, but also for legal reasons having to do with exposing staff
to offensive material.
Eliminating spam can be done in many different ways, and at different
locations in the "email stream". This article limits itself to the
enterprise where the mail comes in. In some cases the ISP may already
have attempted to filter some spam out, using known strategies like
existing blacklists or dedicated "filter companies" that charge monthly
subscription fees, but looking at the amounts of spam still coming
through, it is clear that these solutions are inadequate. The situation
is very similar to anti-virus protection companies that are practically
always in a "reactive" mode. Time for another layer of defense!
So let's have a look at the three techniques that are used at the moment
to filter out junk email at the enterprise level:
Here is a more detailed look at each one:
- Large Community-created filter rules (a la CloudMark/SpamNet/Razor)
- Automated Rules Based filters ("artificial intelligence")
- Permission-based. ("I do not know you, go away & ask permission first")
1 - Community created rules (a la CloudMark/SpamNet/Razor)
This sounds like a nifty idea, until you start thinking about it.
A group of people deciding if the email you get is spam or not??
If there are errors in the execution, this group could quickly
become a censor that plonks emails from your favorite software
company in the spam bucket. Operating System Religious Wars anyone?
Personally I do not feel comfortable with that, and the result is
that you still need to look at all the spam to make sure there are
no false positives (email tagged as spam that isn't). In the case
of CloudMark, its backbone is Razor, a Linux-based open source
solution. The community spam filtration algorithms are openly
documented, leaving the door open for smart spammers to "fool" the
system. Of course, this won't be an issue until CloudMark gets enough
users to actually be a headache for spammers--which ironically would
make CloudMark highly likely a victim of its own success.
2 - Automated Rules Based filters ("artificial intelligence")
Definitely useful, but sometimes lacking sufficient smarts to tag
spam and quarantine it in the junk email folders. The key here is
that you need a regular (ideally daily) update of the rule set.
With these rules kept up to date by humans that apply their
intelligence and common sense this is definitely a good approach.
3 - Permission-based. ("I do not know you, go away & ask permission first")
There are a number of problems with this approach. First,
everyone is assumed to be spam unless pre-approved. Assuming all
senders are spammers unless pre-approved is not a practical method
of spam detection. The reason is simple: what if someone you know
but don't have on the pre-approved list emails you? They would have
to go through the hassle of becoming approved before they can get
a message across. Double or triple the traffic. It happens often
that people email you, and you don't have them in your address book.
They may be an old classmate or the message is a potentially very
lucrative offer from a yet unknown business associate. You definitely
want to get their email without making them jump through the cyber
equivalent of burning hoops.
In a corporate environment, not knowing if an email you hoped to get
was filtered out or not is a disaster. Trying to find out if an email
was sent, if it got filtered or not, why it did not arrive (and the
like) will cost so much time and money in both support staff time and
lost productivity that a good part of email's advantages will be lost.
Just think about the inter-departmental wars that will be fought
over who controls the spam filtering rules.
In short, for a variety of reasons (legal not being the least) the
end-user needs to be the sole judge regarding the email they receive.
The ideal situation at the end-user level is to have powerful rules-based spam filtering combined with an easily manageable pre-approved
"whitelist", and also an end-user controlled "blacklist" where they
can plonk mail from senders they deem timewasters. From an "internal
politics" perspective, this is by -far- the most appealing approach.
And the number one concern of the end-user ("I might not receive
that important email") is also solved this way.
Now that we have established what the best method to filter spam is,
next comes the question how to implement it? And here again are some
quite different approaches.
And what is the best way?
- Email Filters at the server level
- Separate Proxy Code that sits between the email client and the server
- Outlook Add-ins that integrate with the email client itself
Server-based filters may technically be a good solution, but for HR,
legal, helpdesk support and internal politics reasons they could
backflash as they may be considered censorship. Not the ideal approach.
Consider this scenario: You go out and purchase an Exchange filter.
However, one of your marketing managers gets a "weekly marketing
tips" newsletter -- and suddenly finds it removed. Or, on a more
mundane (and human level), one of your people in Accounting just
loves her daily horoscope. While it is difficult to argue the ethics
of getting a daily horoscope at work, there is the reality that a
user signed up for particular information and would still like to
A separate proxy running on the client is difficult to maintain.
Furthermore, most of these applications only work with POP, which
makes them incompatible with Exchange server environments. Finally,
in the corporate environment, more applications, more code and more
ways things can break is not a good idea. Moreover, having to enter
passwords during installation to communicate with the mail server
is not an option for thousands of users: it's a recipe for problems.
Tools like McAfee's SpamKiller work this way only because they have to. Compatibility with many different email clients forces that
kind of cumbersome architecture.
Add-ins for Outlook are small, so called "next-next-next-done"
installs, and smoothly integrate in a known environment. The
learning curve is close to nothing and no email gets lost ever.
It just gets quarantined. No support calls, no complaints about
mail not received, no pain installing another client app, and no
pain in the budget because for large volume licenses the cost per
user is a few bucks and the ROI is probably 5 days. Now here is a
chance for corporate IT departments to change their image from the
"network nazi" to that of the hero that eliminated annoying and
time-wasting junk email by treating the end-user as a grown-up and
empower them to control their own inbox.
Above reasons are what made us design the iHateSpam product the
way we did: