A long time ago, when I was first introduced to email, I was using the Mail program from Unix. I quickly converted to Elm, then Mutt, which were both better in terms of interface. Then I found out about Gnus, and I wouldn't dream of letting it go now. However, Gnus has started showing its age several times, and several times have I needed to upgrade the way I was using it: first because I needed to sort and split email, then because I took the sorting out of Gnus and into Procmail for more advanced filtering (including spam filtering), then because I switched to storing the emails on an IMAP server so I could read them remotely from several computers. My setup as of a few days ago was functional, but since I have grown over time to splitting emails into several hundred folders, checking for new messages was becoming more and more boring.
So it's time to jump in with all the cool kids and switch to a modern solution: still Gnus of course, but with Dovecot, OfflineIMAP for synchronisation, and let's add email searches into the mix while we're at it. My web searches didn't turn up a simple step-by-step HOWTO, but I assembled bits from different places, and here's my attempt at documenting my new setup.
Goals
- Gnus
- Offline operation
- Searching through emails
- …and make it fast!
Assumptions
- Email gets delivered (by SMTP or fetchmail or whatever) to a remote server, which can be accessed through IMAP.
- The client is running some sort of standard Unix-like system; Debian GNU/Linux for me, but it should also work with a BSD or Solaris or something else.
- There is enough space on the client to have a copy of the whole email store (or at least of the folders you want to use with this setup — Gnus can of course access the others through the slower IMAP server).
Dovecot setup
We'll use Dovecot as a local IMAP server. And since we're lazy, we'll access it over a pipe, and dispense with the network part.
- Install Dovecot (
aptitude install dovecot-imapdon Debian systems and related). - Disable its automated running unless you need it for other purposes;
on Debian, set
ENABLED=0in/etc/default/dovecot. - Decide on where you'll store your emails locally; typical place
would be
$HOME/Maildir. - That's it!
OfflineIMAP setup
OfflineIMAP is basically an optimised two-way synchronisation mechanism between two email “repositories”. We'll use it in IMAP-to-IMAP mode.
aptitude install offlineimap, or similar.- You'll need a
~/.offlineimaprcfile, with contents based on the following example:
[general] accounts = MyAccount pythonfile = .offlineimap.py [Account MyAccount] localrepository = LocalIMAP remoterepository = RemoteIMAP # autorefresh = 5 # postsynchook = notmuch new [Repository LocalIMAP] type = IMAP preauthtunnel = MAIL=maildir:$HOME/Maildir /usr/lib/dovecot/imap holdconnectionopen = yes [Repository RemoteIMAP] type = IMAP remotehost = mail.example.com remoteuser = jsmith remotepass = swordfish ssl = yes nametrans = lambda name: re.sub('^INBOX.', '', name) # folderfilter = lambda name: name in [ 'INBOX.important', 'INBOX.work' ] # folderfilter = lambda name: not (name in [ 'INBOX.spam', 'INBOX.commits' ]) # holdconnectionopen = yes maxconnections = 3 # foldersort = lld_cmp
- You will of course need to replace the
remotehost,remoteuserandremotepassvalues with your own. If you want to synchronize only a subset of the IMAP folders, uncomment and adapt one of thefolderfilterlines; the lambda can be any Python code you like, I guess. - Run
offlineimap. See the emails coming in and being replicated to your local store. - Run
MAIL=maildir:$HOME/Maildir /usr/lib/dovecot/imap. You should get* PREAUTH [CAPABILITY ... STATUS] Logged in as your-login. Type* LIST "" *in there. You should see a list of all the folders. - The
.offlineimaprcmentions a.offlineimap.py, which is where we're going to store some additional code used by OfflineIMAP. Here's a sample:
# Propagate gnus-expire flag from offlineimap import imaputil def lld_flagsimap2maildir(flagstring): flagmap = {'\\seen': 'S', '\\answered': 'R', '\\flagged': 'F', '\\deleted': 'T', '\\draft': 'D', 'gnus-expire': 'E'} retval = [] imapflaglist = [x.lower() for x in flagstring[1:-1].split()] for imapflag in imapflaglist: if flagmap.has_key(imapflag): retval.append(flagmap[imapflag]) retval.sort() return retval def lld_flagsmaildir2imap(list): flagmap = {'S': '\\Seen', 'R': '\\Answered', 'F': '\\Flagged', 'T': '\\Deleted', 'D': '\\Draft', 'E': 'gnus-expire'} retval = [] for mdflag in list: if flagmap.has_key(mdflag): retval.append(flagmap[mdflag]) retval.sort() return '(' + ' '.join(retval) + ')' imaputil.flagsmaildir2imap = lld_flagsmaildir2imap imaputil.flagsimap2maildir = lld_flagsimap2maildir # Grab some folders first, and archives later high = ['^important$', '^work$'] low = ['^archives', '^spam$'] import re def lld_cmp(x, y): for r in high: xm = re.search (r, x) ym = re.search (r, y) if xm and ym: return cmp(x, y) elif xm: return -1 elif ym: return +1 for r in low: xm = re.search (r, x) ym = re.search (r, y) if xm and ym: return cmp(x, y) elif xm: return +1 elif ym: return -1 return cmp(x, y)
The first part of this file adds a new flag that OfflineIMAP will
propagate back and forth. By default, only the standard IMAP flags
are propagated; we also want to synchronize the gnus-expire flag
that Gnus uses to mark expirable articles. It's a hack, but it works
for now (maybe someday OfflineIMAP will propagate all the flags it
finds?).
The second part of that file can be dispensed with (and won't be used
unless the foldersort option is uncommented in .offlineimaprc):
it's only there to ensure that some important folders are propagated
first, and some others go last. I don't know exactly how they are
sorted by default, but I'd like the most important ones to come first,
so I can start reading them while the archives and the spam are still
being fetched.
Gnus configuration
- You need to set up a “select method” for the local IMAP server. The
method will be
nnimap, the address can be whatever you like (it's just a name anyway), and you only need two options:nnimap-stream, set toshell, andnnimap-shell-program(orimap-shell-programin older versions of Gnus), which you set to"MAIL=maildir:$HOME/Maildir /usr/lib/dovecot/imap". - Then browse the server, subscribe to the folders, and so on.
- From time to time, run
offlineimapfrom a terminal, so you get the new messages, delete the old ones, and so on. - What's that? You hate running stuff by hand? So do I. To run
OfflineIMAP in a loop, uncomment the
autorefreshoption in.offlineimaprc. The value is the duration (in minutes) between two runs. - Even starting OfflineIMAP is too much work?
offlineimap.elto the rescue! Grab it, put it where Emacs will find it, and add the following to your.emacs:
(require 'offlineimap) (add-hook 'gnus-before-startup-hook 'offlineimap)
Bonus: email searches
The simple way:
(require 'nnir)
This goes in your .gnus. Then your group buffer will get a new
command. Mark some folders with #, then M-x
gnus-group-make-nnir-group (or use the G G shortcut), and type in a
set of keywords. This search is performed by the IMAP server
(Dovecot), which may or may not be very efficient, especially if you
select many folders.
Bonus+: email searches, faster
The real cool kids use Notmuch nowadays, at least for email indexing
and searching. It's fast, it allows complex queries, and it's
generally cool. The downside is that it uses up quite some disk space
for its indices, in addition to the actual emails. For that reason
I'll keep it to my main computer, and I'll stick to nnir on my
laptop (which has the same setup apart from that).
aptitude install notmuch, yada yada.notmuch setup, tell it where you stored your Maildir. If you're only going to use Notmuch for searches, I suggest settingtags=to an empty value in.notmuch-configafterwards. We don't want Notmuch to intrude.notmuch newwill get it indexing the messages you already have.- Of course, being lazy is being cool, so let's uncomment the
postsynchookline from.offlineimaprc. - Now add
(require 'notmuch)to your.gnus. Also(define-key gnus-group-mode-map "GG" 'notmuch-search), for the shortcut. - But seeing just the message that matches is not enough, sometimes we
want the whole thread. Here's a snippet of Lisp for your
.gnus, based on Tassilo Horn's configuration. Do a Notmuch search, enter one of the results, typeC-c C-c, you'll get transported to the folder where that message was, with the context. Note this requires code fromorg-mode, so you might need to install that.
(require 'notmuch) (add-hook 'gnus-group-mode-hook 'lld-notmuch-shortcut) (require 'org-gnus) (defun lld-notmuch-shortcut () (define-key gnus-group-mode-map "GG" 'notmuch-search) ) (defun lld-notmuch-file-to-group (file) "Calculate the Gnus group name from the given file name. " (let ((group (file-name-directory (directory-file-name (file-name-directory file))))) (setq group (replace-regexp-in-string ".*/Maildir/" "nnimap+local:" group)) (setq group (replace-regexp-in-string "/$" "" group)) (if (string-match ":$" group) (concat group "INBOX") (replace-regexp-in-string ":\\." ":" group)))) (defun lld-notmuch-goto-message-in-gnus () "Open a summary buffer containing the current notmuch article." (interactive) (let ((group (lld-notmuch-file-to-group (notmuch-show-get-filename))) (message-id (replace-regexp-in-string "^id:" "" (notmuch-show-get-message-id)))) (setq message-id (replace-regexp-in-string "\"" "" message-id)) (if (and group message-id) (progn (switch-to-buffer "*Group*") (org-gnus-follow-link group message-id)) (message "Couldn't get relevant infos for switching to Gnus.")))) (define-key notmuch-show-mode-map (kbd "C-c C-c") 'lld-notmuch-goto-message-in-gnus)
Wrap-up
This setup can be replicated on several computers, of course. I have it on two, and there's no reason I couldn't have more. The flags do get propagated back and forth, including the Gnus-specific “expirable” flag. Accessing the local Dovecot is much faster than going through the DSL to the “master” IMAP server, and I'm pretty convinced that OfflineIMAP and its multi-threading is also faster than Gnus is, even talking to the same remote server. The email searching with Notmuch is a nice bonus, especially since they're fast too (and this despite my 8-year-old computer).
There are a few minor glitches. I can live with them, but I should let you know anyway.
offlineimap.elkeeps an OfflineIMAP process running (if usingautorefresh), which causes Emacs to complain about when you want to exit.- OfflineIMAP propagates the new folders from the remote server to the local store, but not the other way round. Folders created from Gnus won't be propagated, so they'll only be visible on one computer.
Apart from that, I'm pretty happy with this new setup. So I hope this
documentation will be useful to others, so I can spread the happiness
around. Send your thanks to the authors of the software involved
(Gnus, Dovecot, OfflineIMAP, offlineimap.el, Procmail, and so on).
Let's see how many years I'll keep that system!
Update: I'm told that starting with version 2.0 of Dovecot, the
correct way to tell the server where to find the mail is something
like /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/Mail
rather than MAIL=maildir:$HOME/Maildir /usr/lib/dovecot/imap. If
you're using that version, you probably need to make the change (both
in OfflineIMAP's and Gnus's configuration).
