On Wednesday I was in Edinburgh to present a talk on spam, backscatter and why traditional backup MXs aren’t a good idea.
Thanks to Techmeetup for inviting me to speak. The video is now online at vimeo.com.
I’ve added individual country zones to georbl.info. If you’re only interested in one zone, or are using it with something that doesn’t support TXT lookups (postfix without patching?) you can perform lookups with less hassle.
If anyone has some config examples for other MTA’s, or knows if postfix will support TXT lookups, please let me know. Otherwise, I’ll write something up later myself.
We’ve had a couple of customers who want to be aggressive on spam, *but* don’t want to risk losing any business emails, however broken the mailserver that it originates from.
The oil industry seem to be particularly bad, and having two marketing companies using our service and a chain of casinos also make for fun times when using various filters.
A couple of months ago I implemented some tighter spam controls. Basically, enforcing the RFCs a bit more tightly because we know spammers take short-cuts. Most of these controls are still in place, but I’ve had to exempt several of our customers due to complaints that email wasn’t getting through. It seems it’s not just spammers that take short-cuts - there are a lot of amateur mail admins out there, and we’re not just talking cowboys who’ve thrown an M$ Exchange server in without taking it out of its cellophane. We’re talking BIG companies (lots in the oil industry), technical companies, all sorts.
You’d think being strict with enforcing RFCs would be reasonably safe, but I’ve lost count of the number of mailservers that don’t have a postmaster address set up, that send from invalid addresses, don’t have reverse IP resolution set up etc. etc. etc. These are really good ways to catch out spammers at smtp time, but from time to time it catches a real email and I’m tired of explaining to customers that it’s the other guy’s mailserver that’s broken.
Many email RFCs have been broken, bent and ignored for so long that suddenly enforcing them breaks things.
Rejecting mail at SMTP time is the “right” way to do things. It reduces bandwidth, memory, cpu and disk usage and eliminates backscatter. In a large ISP the two main costs are power and bandwidth, and so there are real cost savings to be made by enforcing RFCs at SMTP time. It’s even good for the environment. By ruthlessly checking for a postmaster address I know that while I sit at my keyboard here, I’m doing my bit for the polar ice caps.
By fortunate coincidence, the most problematic of our clients *only* receive email from UK companies + a couple of known addresses that we can whitelist individually. So, if we could whitelist *everything* from the UK as well, we’d be pretty sure of not missing and valuable emails.
I’ve taken an old script of Dan Shearer’s (thanks Dan) for grabbing the IP ranges from RIPE, APNIC, AFRINIC, ARIN & LACNIC, updated it and hacked it around so it spits out zone files suitable for use with rbldnsd. If anyone else wants to make use of it, feel free. http://georbl.info/
When testing routing behavior in Exim, remember to flush the callout cache. If an address callout has failed, that failure will be cached to speed up routing should another email arrive for that address. This is normally a good thing, but a pain if you’re testing configs because you’ll keep getting the address rejected even if you’ve fixed the routing problem.
rejected RCPT <joe@bloggs.com>: Previous (cached) callout verification failureThere are two solutions.
The first is to turn off callout caching using the no_cache option in exim.conf. eg:
verify = recipient/callout=no_cacheor you can flush the callout cache. Simply -HUPing or restarting exim won’t do. You need to delete the cache manually:
cd /var/spool/exim/db
mv callout callout.deleted # or just delete it
Google “freeserve spam” or “wanadoo spam” and you’ll see these guys get themselves blacklisted frequently by dns blacklists for backscatter spam and for spammers using their network. Spamhaus, Spamcop and Sorbs all pick up some of their servers regularly. Their mails get bounced by any email provider using dns blacklists. Unfortunately, people keep using them. Why?
I’d write more on this, but so many have already.
If you’re attached to your Freeserve/Wanadoo/Orange account, but are sick of having your emails bounced, I suggest reading Andrew West’s Gmail workaround for Orange/Freeserve/Wanadoo customers with email problems.
One of the domains I host has recently attracted a lot of backscatter spam. What is backscatter? Let me explain.
If a spammer fakes an email address on someone else’s domain, some incorrectly configured mailservers receiving the spam will bounce the message back to the (apparent) sender. Meaning whichever poor schmuck has had their domain faked will get a huge pile of bounce messages. Thousands. Some misguided email administrators will even ban email from the domain and/or mailserver that appears to send the message.
Despite the abundance of information available on backscatter spam, there are still loads of mail servers that will happily bounce mail in this manner. Exim, Postfix and Sendmail, configured correctly, are all capable of dealing with this problem. If you’re looking for a reason to avoid using qmail, backscatter spam would be a good place to start.
The only time a mailserver should reject a message is at SMTP time. ie. when it is still connected to the sending machine. Once a mailserver has accepted an email for delivery it has made a commitment to deliver the email. Therefore a mailserver should never accept email that it isn’t able to deliver. Bounce messages should only be sent to local clients to indicate that their message did not get through.
The versions are coming thick and fast. Seems I’m not the only one who was busily working through the holidays. This time its listed on the the exim site. A few bugfixes from 4.64 & 4.65.
I’m up to my elbows in various projects at the moment, rebuilding my MTAs being one of them. I’m converting my old sendmail/postfix MTAs to exim and separating out the functions onto different Xen virtual machines. Where possible, for maintainability, I like to stick with pre-rolled Debian packages unless I have a reason not to do so. Exim is one case where I like to roll my own.
Much of the exim config is best built into the binary at compile time. One example is logging. Exim needs to know where to write logs before it reads its config (in case it needs to log the fact that it can’t read its config). If you want to log to a non-standard location (or use syslog), then you need to tell exim at compile time. You will also need to compile from source if you want to specify which user exim runs as or if you want to compile out unused transports and lookups for security/performance reasons.
Fortunately, exim is pretty easy to put together, and local makefiles are mostly transferable from one version to the next so its easy to maintain your own package. I run three separate daemons for the external mail exchanger (mx), local delivery (mailstore) and outgoing smtp. Building and maintaining these is straightforward - I’ll be putting a how-to up on this site in the next week or so.
Philip Hazel announced a new release of Exim a week ago:
The Exim website still says 4.63 is the current version, but the mirrors seem to have 4.64 in stock.
I’ll get busy building a deb package for Xen/Debian Etch.
I notice Google has picked up this blog recently so I guess I’d better start writing in it. Drafts of various articles have been underway for a while, but I’ve had little time to finish them. I expect I’ll be writing more over the Christmas break. Last year I spent considerable time testing and reviewing Open Source LAMP apps between eating various roasted animals and consuming vast quantities of alcohol. Bliss.
Gladserv.com is a step or two closer to being launched as a business hosted services provider. The domains are registered, the website is coming together, the second dedicated server has been ordered from Bytemark. An earlier order from UK2 was aborted when I discovered just how difficult they were to contact. Take a look at “Why Not To Use UK2” if you’re seriously considering them - cheap has more than one meaning. This server will be split into several virtual machines (VMs) using Xen with unused VMs sold off - there are already three other businesses on board.
I went to see the bank yesterday and my bank manager actually told me he thought my revenue estimates for the first year were very conservative. I tend to estimate on the side of caution these days, after previous bitter experience. This project is definitely gathering momentum.
I’m starting to promote the site. At the speed Google moves, I think it best to link first and write afterwards. I’ve put the shell of the site together using Website Baker, which is probably the easiest Content Management System (CMS) to set up and use I’ve come across. Graphics and pretty stuff will follow when someone with more visual talent than I provides them.
For the moment I have no need for the kind of fancy frippery that something like Joomla has built in. I usually spend the first hour on a new Joomla site turning everything off. For a simple business site, Website Baker has everything needed to get off the ground without additional distractions. There are some addons available to perform most commonly required functions, but nothing like the bewildering range of Joomla toys. Maybe later.
Yesterday I bought an incoming phone number from Gradwell and pointed it at an old Asterisk installation on my backup server. I’ve never used Gradwell for VOIP services before, but they came highly recommended to me so I thought I’d try them out. I’ve had less success with some other providers in the past. No problems at all so far. Online signup was straightforward. At one point I needed to phone for an authorisation code. At 1730 they answered the phone within a few rings and dealt with it on the spot. Provisioning of the line was immediate.
Asterisk setup is a topic for another day, but to add a new number into an existing setup is trivial. Add a few lines like this to iax.conf:
[08708618861]
type=user
username=myusername
secret=mypassword
context=iax-in
host=dynamic
and a line in extension.conf to tell asterisk where to direct incoming calls:
[iax-in]
exten => 08708618861,1,Goto(gladserv,s,1)
Easy. No need to get a man in at all.