Thanks to Techmeetup for inviting me to speak. The video is now online at vimeo.com.

On our outgoing SMTP servers, exim was queuing up messages and failing to deliver them with no clear reason why. It would get partway through the data phase and then the connection would drop. Size did not appear to be a factor.

Eg.

Connection timed out: SMTP timeout while connected to
smtpin.blueyonder.virginmedia.com [62.254.123.242] after sending data
block (98268 bytes written)

The problem is also described here:

http://www.mail-archive.com/exim-users@exim.org/msg29283.html

Initially I was thrown off the scent, as this *only* affected my newer outgoing smtp servers that use standard debs. The affected servers were in different datacenters, with no obvious common link. My old hand rolled servers were unaffected. My temporary hack was to attempt direct delivery, and if it failed, pass it to my old (t)rusty hand-rolled servers for delivery. This worked. For months.

Until the other day. Nagios alerted me that we had some messages queued.

When I retired an old Xen dom0 I picked up one of the unaffected smtp servers and migrated it to a new Xen host. Suddenly it started exhibiting the same problem. The Xen dom0 was 3.3 instead of 3.0, networking had changed from bridged to routed, and I’d installed grub and a kernel to boot it under pygrub. Nothing else had changed.

A friend suggested it must be network related. I figured he was probably right, as that was the main thing that had changed, but what? So I googled: xen networking problems exim sending email and found my own blog entry on tx checksumming!

  ethtool -K eth0 tx off

Problem solved.

It seems these servers were created before that became part of our standard build, and it had just never caused a problem on the old hardware/network setup. Argh.

I think it’s about time I wrote my post on Monitoring Driven Infrastructure to explain why this *should* never happen.

CPU, disk I/O and memory usage were all minimal on both servers and on the host, and yet the LAMP application was performing like a dog with one leg.

Fortunately I’ve seen this before, and (after banging my head on my keyboard a few times) I recognised the problem. Xen networking.

Basically, when the operating system sends a packet through the network, it computes a checksum of the data so the recipient can tell if it arrived intact. On a physical machine, the operating system offloads this operation to the network card as its chipset can handle it. However, in a virtual machine, there is no physical card to hand the checksumming off to - the network card is just an abstraction in software, and so this is terribly inefficient.

This problem only affects two virtual machines on the same hardware talking over the virtual network. [correction: Actually, it affects all network traffic, it’s just more noticeable between two adjacent VMs] So, not usually a problem, but when the application server needs to talk to it’s database server and they both happened to be on the same hardware it makes a huge performance difference.

To fix, use ethtool. First, check the settings (do this on each domU):


# ethtool -k eth0
Offload parameters for eth0:
Cannot get device rx csum settings: Operation not supported
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off


TX checksumming is on. Turn it off:

ethtool -K eth0 tx off

And verify the result:


# ethtool -k eth0
Offload parameters for eth0:
Cannot get device rx csum settings: Operation not supported
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off


I figured this out before, and fixed it, but when I rebooted the server the change was lost. I’ve now fixed it permanently in /etc/rc.local (you could also do this in a post-up network script, although rc.local will run after networking is up anyway).

I’ve already put this into the configuration management system (we’re using puppet - I’ll make that the topic of another post), but these are some old VMs that are not yet automated.

So - fixed. And the servers are humming along even better than before now after a hardware upgrade.

One of the projects we became more involved in was FixTheBuses. They needed an SMS message interface to be able to send and receive text messages from people at bus-stops. We managed to get that going (thanks to Anand Kumria for help here), and the rest of the team put together a system that actually works.

I’m really pleased to be able to support a venture like SICamp, and I hope we can be involved again.

A short video of the weekend is now online.

My hosting company, Gladserv, was shortlisted in 2008 with the award for Open Source Excellence going to Edinburgh’s Wolfson Microelectronics for their contributions to the Linux kernel.

This year I’ve been invited to join the judging panel, and I look forward to seeing some quality submissions for this year’s Awards.

If you know any companies in Scotland who have made remarkable contributions to F/L/OSS, please nominate them for an award.

My company, Gladserv was shortlisted for the Open Source Business Excellence Award, which went to Wolfson Microelectronics. Fair enough - amongst other things they have a full time Linux kernel developer on staff!

What I found interesting was that none of the shortlisted entries for the Open Source Business Excellence Award were software companies. Two are IC companies: National Semiconductor and Wolfson. And then there’s Gladserv, a Hosted Services and Infrastructure company. We all use and develop software in our business, but software is not our business.

This got me thinking. Gladserv doesn’t publish a lot of code, which isn’t itself a problem. However, we publish a lot less code than we write. Our arrangements with our clients mean that everything we write is available under the GPL, but that’s not a lot of good if it never makes it onto an ftp server. We need to do better, and in 2009 I plan to make sure we do. Along with testing and documentation, publishing needs to become a standard part of our development process.

If anyone has some config examples for other MTA’s, or knows if postfix will support TXT lookups, please let me know. Otherwise, I’ll write something up later myself.

The oil industry seem to be particularly bad, and having two marketing companies using our service and a chain of casinos also make for fun times when using various filters.

A couple of months ago I implemented some tighter spam controls. Basically, enforcing the RFCs a bit more tightly because we know spammers take short-cuts. Most of these controls are still in place, but I’ve had to exempt several of our customers due to complaints that email wasn’t getting through. It seems it’s not just spammers that take short-cuts - there are a lot of amateur mail admins out there, and we’re not just talking cowboys who’ve thrown an M$ Exchange server in without taking it out of its cellophane. We’re talking BIG companies (lots in the oil industry), technical companies, all sorts.

You’d think being strict with enforcing RFCs would be reasonably safe, but I’ve lost count of the number of mailservers that don’t have a postmaster address set up, that send from invalid addresses, don’t have reverse IP resolution set up etc. etc. etc. These are really good ways to catch out spammers at smtp time, but from time to time it catches a real email and I’m tired of explaining to customers that it’s the other guy’s mailserver that’s broken.

Many email RFCs have been broken, bent and ignored for so long that suddenly enforcing them breaks things.

Rejecting mail at SMTP time is the “right” way to do things. It reduces bandwidth, memory, cpu and disk usage and eliminates backscatter. In a large ISP the two main costs are power and bandwidth, and so there are real cost savings to be made by enforcing RFCs at SMTP time. It’s even good for the environment. By ruthlessly checking for a postmaster address I know that while I sit at my keyboard here, I’m doing my bit for the polar ice caps.

By fortunate coincidence, the most problematic of our clients *only* receive email from UK companies + a couple of known addresses that we can whitelist individually. So, if we could whitelist *everything* from the UK as well, we’d be pretty sure of not missing and valuable emails.

I’ve taken an old script of Dan Shearer’s (thanks Dan) for grabbing the IP ranges from RIPE, APNIC, AFRINIC, ARIN & LACNIC, updated it and hacked it around so it spits out zone files suitable for use with rbldnsd. If anyone else wants to make use of it, feel free. http://georbl.info/

The Scottish Open Source Awards opened today, 1st August 2007 at 9AM, for nominations and entries. The awards are open to business, government, education, not-for-profit, charities and students, who contribute to or use Open Source Software or services.

Press Release

Scottish Open Source Awards website.