gladd - a simple httpd server / RESTful API engine

June 3rd, 2013

While I’m publishing code, I suppose I should put some of mine out there.

https://github.com/brettsheffield/gladd

This is a very simple httpd, written in c, which was built to act as the RESTful API back end to the open source accounting software project I’m working on[1]. It’s been pretty stable since about February, with only minor changes since then.

[1] It’s called Gladbooks, and I’ll publish that too, when it’s cooked.

should - an inotify-based replication and synchronisation daemon

June 3rd, 2013

Thanks to Claudio Calvelli, for allowing me to publish his code where others may benefit:

https://github.com/gladserv/should

I met Claudio back in 2008 when we were both working on a Xen virtualisation project for a certain large frozen food retailer in the UK. At this point I’d been looking into better options for real-time WAN replication of files and found the available tools lacking. I described the problem to Claudio, and only a few days later he had a working prototype. It is now countless hours and about 27K LOC later. For the last few years it’s been running happily in three companies that I know of and we find it very useful for, among other things, replicating Maildir file structures.

Today, after a few nudges from Dan Shearer, I’ve put it where others may find it useful.

Spam and Backscatter Talk at Techmeetup Edinburgh

August 15th, 2010

On Wednesday I was in Edinburgh to present a talk on spam, backscatter and why traditional backup MXs aren’t a good idea.

Thanks to Techmeetup for inviting me to speak. The video is now online at vimeo.com.

SMTP timeout while connected to <x> after sending data block

December 10th, 2009

This is a problem that had been annoying me for a while.

On our outgoing SMTP servers, exim was queuing up messages and failing to deliver them with no clear reason why. It would get partway through the data phase and then the connection would drop. Size did not appear to be a factor.

Eg.


Connection timed out: SMTP timeout while connected to
smtpin.blueyonder.virginmedia.com [62.254.123.242] after sending data
block (98268 bytes written)

The problem is also described here:

http://www.mail-archive.com/exim-users@exim.org/msg29283.html

Initially I was thrown off the scent, as this *only* affected my newer outgoing smtp servers that use standard debs. The affected servers were in different datacenters, with no obvious common link. My old hand rolled servers were unaffected. My temporary hack was to attempt direct delivery, and if it failed, pass it to my old (t)rusty hand-rolled servers for delivery. This worked. For months.

Until the other day. Nagios alerted me that we had some messages queued.

When I retired an old Xen dom0 I picked up one of the unaffected smtp servers and migrated it to a new Xen host. Suddenly it started exhibiting the same problem. The Xen dom0 was 3.3 instead of 3.0, networking had changed from bridged to routed, and I’d installed grub and a kernel to boot it under pygrub. Nothing else had changed.

A friend suggested it must be network related. I figured he was probably right, as that was the main thing that had changed, but what? So I googled: xen networking problems exim sending email and found my own blog entry on tx checksumming!


  ethtool -K eth0 tx off

Problem solved.

It seems these servers were created before that became part of our standard build, and it had just never caused a problem on the old hardware/network setup. Argh.

I think it’s about time I wrote my post on Monitoring Driven Infrastructure to explain why this *should* never happen.

Xen Network Performance Problem - TX Checksumming

August 19th, 2009

I was stung by this again today. Two virtual servers on the same hardware trying to communicate with each other. One running apache2 & php, and the other a MySQL database server. This happened just after a server reboot.

CPU, disk I/O and memory usage were all minimal on both servers and on the host, and yet the LAMP application was performing like a dog with one leg.

Fortunately I’ve seen this before, and (after banging my head on my keyboard a few times) I recognised the problem. Xen networking.

Basically, when the operating system sends a packet through the network, it computes a checksum of the data so the recipient can tell if it arrived intact. On a physical machine, the operating system offloads this operation to the network card as its chipset can handle it. However, in a virtual machine, there is no physical card to hand the checksumming off to - the network card is just an abstraction in software, and so this is terribly inefficient.

This problem only affects two virtual machines on the same hardware talking over the virtual network. [correction: Actually, it affects all network traffic, it’s just more noticeable between two adjacent VMs] So, not usually a problem, but when the application server needs to talk to it’s database server and they both happened to be on the same hardware it makes a huge performance difference.

To fix, use ethtool. First, check the settings (do this on each domU):


# ethtool -k eth0
Offload parameters for eth0:
Cannot get device rx csum settings: Operation not supported
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off

TX checksumming is on. Turn it off:

ethtool -K eth0 tx off

And verify the result:


# ethtool -k eth0
Offload parameters for eth0:
Cannot get device rx csum settings: Operation not supported
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off

I figured this out before, and fixed it, but when I rebooted the server the change was lost. I’ve now fixed it permanently in /etc/rc.local (you could also do this in a post-up network script, although rc.local will run after networking is up anyway).

I’ve already put this into the configuration management system (we’re using puppet - I’ll make that the topic of another post), but these are some old VMs that are not yet automated.

So - fixed. And the servers are humming along even better than before now after a hardware upgrade.

Social Innovation Camp Video

August 13th, 2009

One of the more interesting things I was involved in over the summer was Social Innovation Camp. In addition to donating a year’s hosting to the winners (MyPolice), Gladserv provided infrastructure support through the weekend. It was barely controlled chaos, but great fun. Esther & I were running about, tunneling people out of the university network, registering domains and creating virtual servers.

One of the projects we became more involved in was FixTheBuses. They needed an SMS message interface to be able to send and receive text messages from people at bus-stops. We managed to get that going (thanks to Anand Kumria for help here), and the rest of the team put together a system that actually works.

I’m really pleased to be able to support a venture like SICamp, and I hope we can be involved again.

A short video of the weekend is now online.

Scottish Open Source Awards 2009 - Open for Nominations

August 12th, 2009

The new Scottish Open Source Awards website is being launched at tonight’s Techmeetup in Edinburgh. Organised by Greg Soper of Sales Agility, Scotland’s CRM Experts, the Awards are now well established with 2009 marking their third year.

My hosting company, Gladserv, was shortlisted in 2008 with the award for Open Source Excellence going to Edinburgh’s Wolfson Microelectronics for their contributions to the Linux kernel.

This year I’ve been invited to join the judging panel, and I look forward to seeing some quality submissions for this year’s Awards.

If you know any companies in Scotland who have made remarkable contributions to F/L/OSS, please nominate them for an award.

Book: Building a Monitoring Infrastructure With Nagios

June 1st, 2009

I was asked today if there were any books on Nagios monitoring. There are a few technical books out there, and they’re usually out of date. The book I’ve found most useful to date was “Building a Monitoring Infrastructure with Nagios” by David Josephsen. It explains far more about the what and the why, rather than the how. Nagios has an excellent online manual, and I’ve found very few technical books that add much to that. This book, however, is worth a read:

Scottish Open Source Awards

November 29th, 2008

Once again, the Scottish Open Source Awards were held as part of the Scottish Software Awards dinner at the Radission SAS in Glasgow. Thanks to Sales Agility for organizing the Awards again for the second year.

My company, Gladserv was shortlisted for the Open Source Business Excellence Award, which went to Wolfson Microelectronics. Fair enough - amongst other things they have a full time Linux kernel developer on staff!

What I found interesting was that none of the shortlisted entries for the Open Source Business Excellence Award were software companies. Two are IC companies: National Semiconductor and Wolfson. And then there’s Gladserv, a Hosted Services and Infrastructure company. We all use and develop software in our business, but software is not our business.

This got me thinking. Gladserv doesn’t publish a lot of code, which isn’t itself a problem. However, we publish a lot less code than we write. Our arrangements with our clients mean that everything we write is available under the GPL, but that’s not a lot of good if it never makes it onto an ftp server. We need to do better, and in 2009 I plan to make sure we do. Along with testing and documentation, publishing needs to become a standard part of our development process.

Georbl: Individual Country DNSBL Zones

November 12th, 2007

I’ve added individual country zones to georbl.info. If you’re only interested in one zone, or are using it with something that doesn’t support TXT lookups (postfix without patching?) you can perform lookups with less hassle.

If anyone has some config examples for other MTA’s, or knows if postfix will support TXT lookups, please let me know. Otherwise, I’ll write something up later myself.