SPF and multiple domain mailserver

We all hate spam, right? I think you all know the annoying emails, but what is worse than receiving emails? How about when emails you send to someone are for some reason detected as spam? This happened to me recently. When it happened the first time, I assumed it was a one-off problem that wouldn’t happen again, but then after some the same thing started happening again.

How to check it myself?

Most people won’t tell the sender if an email mistakenly ends up in their spam folder. They move it to their inbox or just mark it as not spam. When you do get this information from a recipient most of the time it’s with very few details. I always try to ask people to send me the email that was marked as spam back. It is important that they send it back to you “As Attachment” as this is the only way you can make sure the header fields (the interesting part for diagnosing what’s going on) will come back. Of course, this is a challenge in itself, as a lot of people won’t know how to do this or their mail client may not even  give them the option.

If you are lucky and get back the header, you might find header fields in the source of the email explaining the reason for it being detected as spam. Here are some headers you might find in such an email.

X-Spam-Checker-Version: SpamAssassin 3.3.1 (2013-02-17) on mail.receiving-mailserver.com
X-Spam-Status: Yes, score=6.6 required=6.0 tests=NORMAL_HTTP_TO_IP,NO_RECEIVED,
NO_RELAYS,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E8_51_100,RAZOR2_CHECK
STOX_REPLY_TYPE,STOX_REPLY_TYPE_WITHOUT_QUOTES autolearn=no version=3.3.1
X-Spam-Report:
        * 0.2 STOX_REPLY_TYPE STOX_REPLY_TYPE
        * -0.0 NO_RELAYS Informational: message was not relayed via SMTP
        * 0.0 NORMAL_HTTP_TO_IP URI: Uses a dotted-decimal IP address in URL
        * 2.4 RAZOR2_CF_RANGE_E8_51_100 Razor2 gives engine 8 confidence level
        * above 50%
        * [cf: 100]
        * 0.4 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50%
        * [cf: 100]
        * 1.7 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/)
        * 1.9 STOX_REPLY_TYPE_WITHOUT_QUOTES STOX_REPLY_TYPE_WITHOUT_QUOTES
        * -0.0 NO_RECEIVED Informational: message has no Received headers

To keep this short, The X-Spam-Status tells you that this example is spam with a score of 6.6 where everything from 6.0 up is considered as spam. More details about the individual checks and their score can be found in the X-Spam-Report field. Don’t think it will always be this easy! Mail-servers are configured differently and most of them don’t write such detailed reports to the mail header.

When you’re trying to debug such problems, sending them to gmail is a good idea. Their servers usually have a very good detection rate, so if your emails are marked as spam there, you have a good starting point.

Sadly, gmail’s servers don’t write such a nice report as shown above into the email header, but they do run nearly every check you can to detect spam, and that will come in very handy for checking if the changes you make are working or not.

SPF: The Sender Policy Framework

SPF specifies a method for the receiving mail server to verify that the email is sent from a permitted server. The receiving mail server extracts the domain from the senders email. The domain is then checked for SPF records (DNS records of type TXT and SPF). These records specify which host is permitted to send emails for that domain. If they match the address of the server delivering the email, the SPF check is passed, if not, the mail server might (according to the rules) reject the email.

Why does this help to avoid false spam detection? The more you can make sure nobody except your server can send emails from your domain (address forgery) the better other servers can detect if your email is spam or not. In my case, the SPF record helped to get not detected as spam. My guess is that you maybe collect negative score for a passed SPF verification.

How to setup SPF?

For a complete SPF setup you have the sender and the receiver side. The sending side is the easier side as it only requires a couple of DNS records. The receiving side is the more complex side. The receiving mail server has to validate if the sender is permitted. First I will explain how the sender side of SPF works and after this I will go into the receiver side.

SPF originally used TXT records in DNS as there was no special record for it. DNS TXT records are free-form text records with no syntax. A special SPF record type is now supported in most DNS servers, but for compatibility, the TXT record still has to be available as well. The two record types do not differ much in their syntax. If you want to read the complete SPF syntax, check the SPF project website. As you can see in the description on the SPF Record Syntax page, the separate components in the SPF record are called mechanisms. Each mechanism is prefixed with a qualifier. If the qualifier is omitted, the default qualifier (“+” Pass) is used.

The most common SPF record seems to be following:

"v=spf1 a mx -all"

In this SPF record, the “a” mechanism defines that all A records in this domain are permitted to send emails for this domain. The “mx” mechanism defines the same for all the MX records. The “-all” mechanism defines that everything else is not permitted. In most cases, that is what you need if you have a mail server that is just running for this domain.

If you want to go more into details about SPF there are a few nice wizards out there that can help you to compose your SPF record. One web-based wizard can be found at www.microsoft.com/senderid/wizard. Update: Since the SPF wizard at Microsoft does not exist anymore, here is a link to the SPF wizard from Unlock The Inbox. I used Webmin to configure BIND DNS server, and found one issue while doing so.

Managing SPF for a multiple domain mail server

When you have a mail server that is handling multiple domains, like me, you will have to permit your mail server to send emails from all your domains. There are two very interesting mechanisms in SPF helping you to manage this very easily. Your mail server will be running under one domain. That is the domain the mail server is responding with in the SMTP greeting. This is what I would like to call for now your primary domain. The primary domain should have an SPF record similar to the one above. This allows the mail server, which of course is listed in the MX of the primary domain, to send your emails.

If you define an SPF for your primary domain, you should define it for all your domains the mail server sends emails for. It appears to be part of the recipient server’s SPF checks to connect to the domains MX record server and verify the SMTP greeting. If the domain from the SMTP greeting has an SPF record defined, but the other domain has not, you might run into problems. My advice is, do it for all domains or for none.

For the other domains, there is the mechanism “include” that is interesting for easy management of the SPF records of the non primary domains. The “include” mechanism includes the SPF record from another domain into the check. That way you can include the primary domain’s SPF record into the other domain’s SPF record. This allows you to change the SPF rules of the primary domain without the need to update all of the domains as well. The SPF record for the non-primary domains might look like this.

"v=spf1 include:primary-domain.com -all"

When you add this to all your domains and point them to your primary domain, the rules from the primary domain are also applied for the other domains.

As well as for the primary domain, it is important to create the SPF and the TXT record as well for the non-primary domains.

How to verify the SPF record?

As with all configuration changes you should test them to make sure everything works as expected. Often you have a small typo in your configuration that can cause unexpected behaviour, so testing your changes is always important.

There are a number of websites that offer to test your SPF setup by entering a list of information, from email addresses to your mail server’s IP address and more. These seem to work but are awfully complicates to use. I have tried a couple of them, and was annoyed how much you have to enter all the time. Then, eventually I found a solution that was way easier.

The website port25.com provides the easiest way I have seen to check your SPF records. You simply send an email from the domain you want to test to check-auth@verifier.port25.com and they do all the rest. After a couple of minutes you receive a full report. They check way more then just the SPF record; they will perform the following checks for you: SPF, SenderID, DomainKeys, DKIM and Spamassassin. The only thing you have to do is wait a few minutes and read the report that is returned to your email.

A similar solution is available at http://www.brandonchecketts.com/emailtest.php where you receive a random email address to send an email to. After the email is received you can click “View Results” and it will give you a detailed report about DomainKeys, DKIM, SPF and Spamassassin. Some of this information is a bit more detailed then the result received from the previous solution. For example, you get information about which SPF mechanism was matched.

Both tools are very easy and provide you a lot of useful details.

How to implement SPF on the receiving side?

For sending emails protected by an SPF record the above procedure is all we need. When we want to check incoming emails we have to implement the check in the incoming mail server. I am using postfix as MTA (Mail Delivery Agent) where a module exists to perform the SPF checks.

The server I use is running CentOS with the “yum” package manager. So, to install the required Perl module I just run this.

sudo yum install perl-Mail-SPF

If you use another distribution, check your vendor specific software repository to see if they have a package. If they don’t, there is still the option to use CPAN. CPAN (Comprehensive Perl Archive Network) is a repository of Perl modules. To install it from there you need to start the Perl shell.

sudo perl -MCPAN -e shell

Within the Perl shell you can now install the required perl module with just one line.

install Mail::SPF

This should install the Perl module and all its dependencies. It may install additional modules to build the module, if they’re not already installed.

The next package we need is the postfix module from www.openspf.org. The module is actually hosted at launchpad.net and, at the time of writing, the latest version is 2.010. The following commands will download the postfix-policyd-spf module, unpack it and /move it into place.

wget https://launchpad.net/postfix-policyd-spf-perl/trunk/release2.010/+download/postfix-policyd-spf-perl-2.010.tar.gz
tar xvzf postfix-policyd-spf-perl-*.tar.gz
cd postfix-policyd-spf-perl-*
sudo mv postfix-policyd-spf-perl /usr/local/lib/policyd-spf-perl

Now we need to edit the postfix configuration. To be exact, we need to add the following to the end of the /etc/postfix/master.cf configuration file.

# postfix policy SPF module
policy unix - n n - 0 spawn
        user=nobody argv=/usr/bin/perl /usr/local/lib/policyd-spf-perl

In the /etc/postfix/main.cf add “check_policy_service unix:private/policy” to the end of the “smtpd_recipient_restrictions” setting. The order of these elements is important; make sure you specify the “check_policy_service” after the “reject_unauth_destination”, otherwise your mail server can become a so called “open relay”. Your configuration might look like this after you’ve made the changes:

smtpd_recipient_restrictions = permit_sasl_authenticated,permit_mynetworks,reject_unauth_destination,check_policy_service unix:private/policy

Test the SPF policy verification

After all the configuration you need to restart postfix and test what you have changed. You remember the gmail account we used at the beginning? We now can use it to test our server. Send an email from the gmail account to your server and then view the received email source as plain text to make sure you can read all the headers. Within the headers you should find something like I have shown below. The first example shows an email from a domain that has an SPF record defined and the verification has passed, and the second example shows an email from a domain that does not have an SPF record defined.

Received-SPF: pass (gmail.com ... _spf.google.com: Sender is authorized to use ‘username@gmail.com' in 'mfrom' identity (mechanism 'include:_netblocks.google.com' matched)) receiver=mail.example.com; identity=mailfrom; envelope-from=“username@gmail.com"; helo=mail-ea0-f170.google.com; client-ip=123.123.123.123

Received-SPF: none (no-spf-example.com: No applicable sender policy available) receiver=mail.example.com; identity=mailfrom; envelope-from="username@no-spf-example.com"; helo=mail.no-spf-example.com; client-ip=234.234.234.234

The related /var/log/maillog on the server will show the same information. If you want to show only those lines, grep for “postfix/policy-spf”.

Read more of my posts on my blog at https://blog.tinned-software.net/.