Merging and migrating email systems

In Wales then we’ve experienced more than our fair share of mergers between education providers.  In variably one of the initial projects that IT services has to complete is the merger of email systems.  As mergers are usually happening at a fast pace, then there are often tight timescales for the organisations e-mail addresses to change to a new format, reflecting the organisations new entity (domain name).

There are many solutions to this problem; However, the one thing that isn’t usually available is the staffing and time required to find solutions, or to build a new directory service around the new entity, as well as to migrate dependant services, end-users and devices to that new directory.  Whilst this is often the end game, it needs a longer-term approach, maybe two or three years..

Interim solutions are often required to enable the organisation to appear as one entity, whilst maintaining multiple legacy systems behind the scenes.  This allows for longer planning time for the end game above, it should minimise disruption to end users and allow for a relatively rapid deployment.

With that in mind then a few years ago I worked with a pair of FE colleges(s) who were merging and put together an interim solution that we could deploy.  That solution was in place for approximately one year following the merger, and was also used in transitioning to the new directory and e-mail service, which reflected the new entity name. So there is also scope to use this to migrate email systems.

Back in the summer I got the opportunity to try re-deploying the solution, which appears to have gone remarkably smoothly. As a result I’m releasing my code into the wild! It’s not elegant or glamourous but it seems to a job quite well.

The following is a bit technical…. so lets start with a diagram of how the potential final system pans out in terms of delivering mail.

The generate-aliases script is written in bash, and uses ldapsearch, grep, awk, and sed to manipulate the output (all the things you might find in a Linux system).  In both scenarios, it was configured to talk to two or more Microsoft Active Directory/Exchange environments (it could be tweaked to talk to others) and to uses the ldap proxyAddresses field to determine potential addresses.  From there it generates the new address list based on the existing mail address.  In-terms of it’s output then it creates a Linux/Unix aliases file.  You could then use any Mail Transfer Agent/Mail Server, that will run on Linux/Unix.

We have used Exim.  The main configuration points for Exim, is to accept mail as a local_domain for neworg.ac.uk and to route mail for the existing domains orga.ac.uk and orgb.ac.uk to the internal Exchange servers.  To do all this then you will ideally need to have networking/ internal routing between the two organizations (whether direct or over a VPN).

The script deals with duplicates (a problem for anyone merging groups of users) by halting the process and issuing an e-mail alert, these then need to be manually resolved before any further changes can be applied.  That’s the only bit of day-to-day administration and the initial hurdle in getting the system up and running.  One of the key elements to this is communication between the teams managing the different mail systems.

One downside to this script is that it does not populate Active Directory in anyway, so the lack of an e-mail list in the Global Address Book can be a problem for users.  However, it should be possible to engineer an import / export with Active Directory using csvde import or just publish the aliases list internally.

Also ahead of going live, then you will also need to add the new e-mail domain to all users on both AD systems and resolve those initial duplicates.  You will also need to add as an Accepted Domain / Internal relay domain, and when ready to go-live set this as the users Primary address/sender address.

One distinct advantage this script has over other solutions is that it also generates a definitive e-mail list which if your mail filtering solution does any form of recipient verification for the next hop, then it will cut down on the potential for Collateral Spam by ensuring that e-mail is accepted or rejected at the front of house mail server (not accepting and issuing a bounce later which can create Collateral Spam).

Anyway, I hope this is useful to anyone else who might be merging e-mail systems or even in migrating from one email system to another.  Feel free to comment/discuss below or contact me.

Greylisting and Cloud-based mail

If your running Greylisting on your external mail gateways, and you implement one of the Cloud based e-mails for the another domain or sub-domain, then your mail from the Cloud will experience a delay on reaching those on the internal mail system.

There’s a number of Greylisting solutions out there. This particular solution is based on using (or should I say working around it!) For details on Alun Jones Greylisting module – see http://users.aber.ac.uk/auj/spam/. Typically I’ve implemented this on the Exim mail servers, with a few colleges.

The obviously choice would be to configure all the IP addresses of the Cloud-based mail system to either by-pass Greylisting or add them into the Greylisting IPwhitelist table. However, with the way the Cloud-based mail services work then it’s likely that there are a large number of IP address that could then change a lot.

With this in mind then it’s a good idea to implement something called SPF or Sender Policy Framework (bit much to explain SPF here – read http://openspf.org ) for the Cloud-based mail system. Microsoft seem to recommend it by using an include statement to ref to the SPF record for outlook.com, when implementing Live@Edu. Not sure if Google Mail do?

With this in place then actual workaround is quite simple.

Firstly, you need to make sure Exim has SPF support (exim -bV should confirm). If you don’t then you need to add this. For the Gentoo distribution, then you need to add the ‘spf’ USE flag (either globally in /etc/make.conf or for the package in /etc/portage/package.use). If compiling Exim from source then you need EXPERIMENTAL_SPF=yes in your Local/Makefile.

Secondly, you need the following snip before the Greylisting call in your Exim configure file (or in Gentoo exim.conf)

accept sender_domains = xyz.domain1.ac.uk
spf = pass

Other than a restart of Exim, and some testing and then that’s it. (Exim -bh is of course useful if you don’t want to actually generate some e-mail!)