November 03, 2003

Clean email - Part 1

Like many geeks, I find email can be both a friend and a foe. Whilst it is usually an essential part of ones daily life, it can also be a right nightmare to manage and configure such that it's elegant, pleasant to use, and not a spam-infested bucket of crud. I have recently spent time attempting to clear up my setup, and have a personal email setup configured with a smooth and intuitive client.

In this article, I will describe what I have done to acheive this, beginning with email filtering.

My own personal email setup for my domain is running off my own server, and I have full control of everything, from the DNS servers down to the SMTP/POP servers and my email client. This is part of my philosophy of not relying on anyone else for my serving needs. It allows me to have maximum control over how traffic is handled, and means that any faults are handled by me rather than waiting on hold for someone else to get around to it.

My primary software is Sendmail, Procmail, SpamAssassin and Mutt. Sendmail is my mail transport agent, delivering and accepting mail from remote servers, procmail is my local delivery agent, also acting as a mail filter, SpamAssassin picks out the spam, and mutt is my email client. I will not describe how to install each of these packages, but will explain how I have some of them configured. Basic knowledge on the configuration of these packages is assumed.

Global procmailrc

My global procmailrc (/etc/procmailrc) is fairly simple, and it's main purpose is to run messages through SpamAssassin:

# /etc/procmailrc # Global procmailrc for all users DROPPRIVS=yes # Filter through spamassassin :0fw | spamc # Log timestamp of anything flagged as spam - for stats :0hc | /usr/local/bin/ $LOGNAME

The second recipe passes messages through a spam statistics script, which I can access online. I might describe it another time...

Duplicate mail

Occasionally due to either human error or something wrong in between, duplicate mail comes through to me. The following recipe at the top of my procmailrc detects these and discards any mail that has already been seen. It uses the message-id to detect dupes

# Weed out duplicate messages :0Wh: msgid.lock | formail -D 65536 .msgid.cache

List handling

I'm currently subscribed to numerous mailing lists, and need a way to easily manage all of the email from each, numbering a couple of hundred messages per day. To handle all of this, I have a .procmailrc set to scan for messages from these lists, and to put them into different folders. The normal list handling recipe looks something like this:

:0: * ^Sender: owner-nanog /home/bb/Mail/nanog

This says to deliver anything matching the 'Sender' line to be delivered to the given mailbox. This works for just about every mailing list, and one recipe for each list usually suffices.

However many lists prepend the subject of all messages with the listname in square brackets. This usually helps people identify mail from a list. The problem I find is that it takes up valuable space in the Subject portion of my Mutt message list. When vieweing messages in a folder, I dont need to be told in every messages that it's for a particular list. The following shows what I'd normally see:

10 19 Jul The Wanderer (1.1K) Re: [MPlayer-users] mga_vid.o + kernel 2 11 19 Jul gabucino@mpla (1.4K) Re: [MPlayer-users] mga_vid.o + kernel 2 12 20 Jul Bartek Jakubs (0.9K) Re: [MPlayer-users] mga_vid.o + kernel 2 13 28 Jul Timo Gerke (1.7K) [MPlayer-users] tv recording problem

The [MPlayer-users] part uses up most of my real estate. What I needed was a way to strip this part of each message for a list, prior to delivering to the folder for that list. I have achieved it with the following recipe:

# Weed out [Listname] prefixes from subject lines # # Subject: [SLUG] Procmail help please # becomes # Subject: Procmail help please :0 * ^Subject:.*\[[^ ]+\]\<+\/.* LSUBJECT = "$MATCH" LISTSUBJECT = "$LSUBJECT" } :0 * ^Sender: { :0hf | formail -I "Subject: $LISTSUBJECT" :0: /home/bb/Mail/slug-chat }

The first recipe extracts the interesting part of the Subject header, and the second recipe uses to rewrite this header. The message is then delivered to the mailbox, sans [ListName] chunk. It also preserves the 'Re: ' part of subjects. The first recipe is only needed once in the whole procmailrc, but the second recipe is needed once per list. I now have messages looking much cleaner:

15 N 3 Nov Samuel Kvasni (2.8K) Re: mplayer screen snapshots 16 N 3 Nov Etienne SANDR (0.7K) vsync not working 17 N 3 Nov Joonas Koivun (0.7K) Cut the subtitles 18 N 3 Nov D Richard Fel (0.6K) black and white DVD

Replies that I send back to the list will have the prefix added again by the list software, so other users will not see anything different.

I am also subscribed to some mailing lists that have sufficiently small volume or interest to warrant putting them into my main mailbox. But to help differentiate them, I want to have them coloured slightly differently than the rest of my mail. To do this, I have the following recipes:

# Common list managers get a header added in to assist mutt # Mailman * ^X-Mailman-Version { :0hf * ^List-Id: \/.* | formail -A "X-ListName: $MATCH" } # Yahoo :0 * ^X-Mailer: Yahoo { :0hf * ^Reply-To: \/.* | formail -A "X-ListName: $MATCH" } # Majordomo :0 * ^Errors-to:.*owner { :0hf * ^Sender: \/.* | formail -A "X-ListName: $MATCH" }

These recipes spot common list manager email headers, and to these messages it adds a header X-ListName. The content of this header isn't too important though. I then have my email client apply slightly different formatting to any email which is in my primary inbox, that has X-ListName in the header.

Spam handling

As part of my spam filtering, I have setup a special email address. Any email sent to this address is fed immediately into the bayesian classifier. I use this as a fast way to "fix" filters for any spam that gets past SpamAssassin. This rule is:

:0 * ^TO.* | sa-learn --spam --single

I have configured a keybinding in Mutt so that pressing "S" on a message causes it to be bounced immediately to

Finally, anything flagged by SpamAssassin is tossed into a junkmail folder. I plan on eventually massaging all of this spam for some interesting statistics:

# All spam to the junk mail folder for entertainment :0: * ^X-Spam-Level: \*\*\*\* /home/bb/Mail/junk

These simple procmail rules make a perfect partner with my Mutt setup and help to make email a much more pleasant part of daily schedule. My Mutt configuration is described in a later article.

Posted by Ben at November 3, 2003 11:29 PM
Post a comment

Remember personal info?

Recent Entries
- Linux on an HP Compaq NC6400
- Sodium in water? Bah..try Caesium!
- I'm off to Lugradio
- Food for thought...
- Replacing ugly Helvetica fonts in Xorg
Support me...

Email me:
NL time:03:33
Book: Assassini (Thomas Gifford)
Amazon wish list
Search the web