I have been running my own email server for as long as I remember. I guess I don’t like to rely on third parties for my email, so options like Google’s GMail or Apple’s iCloud are really not for me. I have recently migrated my email from an old system to a new one, and I’d like to document the new setup while it’s still fresh in my mind and thus share my experience with the community.

General Overview

My old setup has aged quite significantly and the pain of its maintenance has been steadily on the increase over the last few years. To give you an idea, I was running a Sun Netra t1 with 512MB of RAM and a pair of 16GB hard drives running in a soft RAID configuration with Solaris 10 as an OS and a combination of Sendmail, SpamAssassin, ClamAV, OpenDKIM and Dovecot. While it was all running fine, at times the server was quite slow, plus keeping on top of the updates was becoming more and more of a problem. One weekend I decided to bite the bullet and deploy an all new solution. Please note, all information in this article is based on excellent OpenBSD FAQ and man pages, as well as a blog series written by Chess Griffin. Thanks guys for your help!

My actual setup consists of three servers: one primary mail server and two backup MX servers.

The primary server hosts the incoming email for myself and other users, plus it’s also used as an SMTP server for outgoing mail. As per the diagram below, it runs:

• PF for packet filtering and port redirection on incoming email to spamd
• spamd running in default mode, with greylisting enabled. This also includes running spamlogd to listen to network traffic on a pflog interface to control greylisting.
• spampd and SpamAssassin combo for mail analysis and tagging
• OpenSMTPD for the actual SMTP service (handling both the MTA as well as the MSA functions)
• Dovecot with IMAP and LMTP services enabled for mail delivery to users mailboxes
• For the MSA functions, I’m also running the dkimproxy for outbound mail to sign the email with the correct DKIM key. Note there’s no DKIM proxy running for the inbound email, as the DKIM is verified by SpamAssassin.
• OpenBGPD for instant black-list and white-list updates from the project’s route servers

The backup servers are only used to queue email sent to my domains should the primary server become unreachable. In this article, the primary server will have a hostname mx1.example.com and its public interface will be configured with IP address 192.0.2.2/24 and IPv6 address 2001:db8:1::2/64. Backup MX servers are mx2.example.com [198.51.100.2/24, 2001:db8:2::2/64] and mx3.example.com [203.0.113.2/24, 2001:db8:3::2/64]. Their setup is a good bit simpler, as they’re running:

• PF for packet filtering and port redirection to spamd, similarly to the configuration on the primary server
• spamd running in default mode with greylisting enabled
• OpenSMTPD for the actual SMTP service, but working only really as an MTA.
• OpenBGPd for instant black-list and white-list updates from the project’s route servers

For completeness, my setup consists of two virtual machines: mx1.example.com and mx2.example.com as well as a physical server mx3.example.com. The virtual machines are hosted on VMware ESXi 6.0 (mx1) and ESXi 4.1U3 (mx2). The physical server is another Sun Netra t1 with 256MB of RAM and 16GB hard drive. mx1 and mx2 are running OpenBSD 5.7, while the mx3 is running OpenBSD 4.9 (due to me not having any out of band access to its console, and therefore not really being in a position where I can upgrade it without the risk of losing access to it). All three servers are in three separate physical locations (actually, three different countries) and use different Internet Service Providers. The spamd on each of the mail exchangers above (mx1, mx2 and mx3) is configured to synchronise the database. This means that if an external host tries to send email through one of my three MX’s, the other two will instantly have a suitable entry installed in their local spamdb’s and will handle the incoming connection correctly. This is depicted on the diagram above with a dotted red line.

Primary MX

The first thing I realised when configuring what was going to become the primary MX was that there are many ways to skin a cat. I’ve gone through the man pages as well as some online guides for setting things up and it all kind of made sense, but it still took me a while to fully understand the details of the configuration and what it is exactly that I wanted to have. I’ve finally come up with the following diagram which shows the data flow through the system for both incoming and outgoing mail.

To understand what’s going on there, you need to start in the top-left corner of the diagram above. The three lines are three scenarios for an email to be processed by the server.

Sending email out to remote recipients (MSA)

The blue line is an email that is being submitted by a user to ports tcp/465 or tcp/587 for sending out to a remote recipient. Please note, when I say a local user, I mean a user who has an email account on my server. Here’s what happens with that email:

1. The TCP session first goes through the PF, which allows the traffic to pass to ports tcp/465 and tcp/587.
2. OpenSMTPD is configured to listen on the two ports above and to use SSL and TLS. The user must also authenticate to be allowed to submit an email for sending.
3. After the user authenticates successfully and his email is submitted for sending, it is then forwarded to localhost, port tcp/10027 (using the relay via smtp://127.0.0.1:10027 statement) to the dkimproxy_out daemon, which adds the correct DKIM key and forwards it back to OpenSMTPD to localhost, port tcp/10028.
4. OpenSMTPD is configured to also listen on the loopback interface, on port tcp/10028 and tag all traffic coming in this way with a DKIM_OUT tag.
5. Finally, all email tagged with the DKIM_OUT tag is relayed out to the remote recipients.

Accepting email for local users (MTA)

The other scenario is for accepting email from the outside world sent in to the local users. There are two paths depicted on the diagram above, one starting with a teal line in the middle and the other one with the red line at the bottom of the three possible “ways in”.

1. The email comes in to port tcp/25 and passes through the PF first. Depending on whether the source IP address of the sender is in the spamd’s whitelist, graylist or blacklist, it either goes directly to OpenSMTPD (in the first case) or is redirected by the PF to spamd, which listens on the loopback interface on port tcp/8025. This is possible because of a number of things:
   • spamd maintains two tables also configured as persistent in the PF, and these are <spamd-white> and <spamd>
   • OpenBGPd maintains another pair of tables, also configured as persistent in PF, which are <bgp_spamd_bypass> and <bgp_spamd>
   • finally, there’s an additional persistent table in the PF, called <nospamd>, and which is used for manually whitelisting known good IP sources
2. If the sender’s source IP is in the <spamd> or <bgp_spamd> tables, or it’s not listed in any of the above tables, that email is redirected to spamd. Spamd is then either greylisting (which means it will temporarily reject the email with a suitable SMTP error code to force the remote sender to send the email back and thus verify if the remote is an SMTP compliant application or a spamming script), blacklist (if it determined that the sender is definitely trying to send spam) or whitelist (place the sender in the whitelist in the spamdb, as well as updating the spamdb and the <spamd-white> table – this means upon next attempt, the email sent by this sender will pass through the pf straight to smtpd for processing)
3. In the next step, the email is then forwarded to localhost, port tcp/10025 to spampd, which handles sending it to SpamAssassin for filtering and classification. On completion, the email is forwarded back to OpenSMTPD, to localhost, port tcp/10026
4. OpenSMTPD is configured to listen on loopback interface port tcp/10026 and to tag incoming email on this interface with SPAM_IN. All email with this tag is then accepted for delivery via Dovecot’s LMTP, provided it is sent to one of the domains listed in the <domains> table and to one of the virtual users listed in the <users> table (note both conditions need to be simultaneously met – more on that later), or that the recipient is listed in the <aliases> table.
5. Finally, the email is delivered to the users mailbox using Dovecot’s LMTP process.

Installation of required software packages

At the time of writing, the current version of OpenBSD is 5.7. I have only used what’s available in the official packages for setting up my email servers.

Before we configure anything, we need to install the following:

• dkimproxy
• dovecot
• p5-Mail-Spamassassin
• spampd

The following, although not strictly required, are also useful (and in fact, this article assumes those packages have also been installed):

• bash
• curl

To install these, make sure you either have a /etc/pkg.conf file with a valid installpath or that the PKG_PATH environmental variable is set with the path to the installation source, and then run the pkg_add -iv PACKAGE command, where PACKAGE is substituted with the package name from the lists above.

PF Configuration

As described above, everything starts with the PF. The primary function of the PF is to protect the server from unwanted traffic, however we’re also using its powerful features to send the inbound email to spamd, or directly to smtpd, depending on which table the sender’s IP is listed in.

#
# See pf.conf(5) and /etc/examples/pf.conf
 
# ----- DEFINITIONS -----
 
# Set the table limits to accommodate 200k entries for spamd-bgp
set limit table-entries 400000
 
# Reassemble fragments
set reassemble yes
 
# Return ICMP for dropped packets
set block-policy return
 
# Enable logging on egress interface
set loginterface egress
 
# Allow all on Loopback interface
set skip on lo
 
# spamd sources
spamd_sync = "{ 198.51.100.2, 203.0.113.2 }"
 
# Define ICMP message types to let in
icmp_types = "{ 0, 8, 3, 4, 11, 30 }"
 
# Define tables for:
# - bruteforce sources
# - bgp_spamd_bypass: SMTP connections coming from these IPs bypass spamd
# - bgp_spamd: SMTP connections coming from these IPs are blacklisted
# - spamd: maintained by spamd, used for greylisting
# - spamd-white: maintained by spamd, used for whitelisting
# - nospamd sources (manual file with list of sources to pass directly to smtp)
table <bruteforce> persist
table <bgp_spamd_bypass> persist
table <bgp_spamd> persist
table <spamd-white> persist
table <spamd> persist # updated by spamd-setup
table <nospamd> persist file "/etc/mail/nospamd"
 
# ----- INBOUND RULES -----
# Scrub packets of weirdness
match in all scrub (no-df random-id max-mss 1440)
 
# Drop urpf-failed packets, add label uRPF
block in quick log from urpf-failed label uRPF
block quick log from <bruteforce>
 
# ICMP
pass in quick inet proto icmp icmp-type $icmp_types
pass in quick inet6 proto icmp6 
 
# spamd sync
pass in quick inet proto udp from $spamd_sync to port spamd-sync
 
# EMAIL - IMAP, POP3
 
pass in quick on egress proto tcp from any \
    to (egress) port { imap, imaps, pop3, pop3s } \
    flags S/SA modulate state \
    (max-src-conn 50, max-src-conn-rate 50/5, overload <bruteforce> flush global)
 
# EMAIL - SMTP
 
pass in quick log (to pflog1) on egress proto tcp \
    from { <bgp_spamd_bypass> <nospamd> <spamd-white> } \
    to (egress) port { smtp } \
    flags S/SA modulate state
 
pass in quick proto tcp from { <bgp_spamd> <spamd> } \
    to (egress) port { smtp, submission, smtps } \
    rdr-to 127.0.0.1 port spamd
 
pass in quick proto tcp from any \
    to (egress) port { smtp } \
    rdr-to 127.0.0.1 port spamd
 
pass in quick proto tcp from any \
    to (egress) port { submission, smtps } \
    flags S/SA modulate state \
    (max-src-conn 50, max-src-conn-rate 25/5, overload <bruteforce> flush global)
 
# the rule below is to exempt any MTA that we send email to from spamd
# to ensure replies to our local users are delivered immediately
pass out quick log (to pflog1) on egress proto tcp to any port smtp
 
# ----- ALL OTHER TRAFFIC TO BE DROPPED -----
 
block in quick log on egress all
 
# ----- OUTBOUND TRAFFIC -----
 
pass out quick on egress proto tcp from any to any modulate state
pass out quick on egress proto udp from any to any keep state
pass out quick on egress proto icmp from any to any keep state
pass out quick on egress proto icmp6 from any to any keep state

The config above is an actual PF configuration file from my host, although sanitised and with some rules not relevant to this article removed. This config is placed in /etc/pf.conf. Please note, that by default, OpenBSD’s PF evaluates rules top to bottom and for any packet matching multiple rules, the last one is going to be applied. While this is a perfectly valid mode of operation, I’m used to configuring firewalls where it’s the first match that is used, and therefore most if not all rules I have in my config use the word quick to implement that behaviour.

First, we configure global settings for PF like raising the limit of table entries to accommodate large BGP tables, enabling fragments reassembly, etc. Note that we also tell PF not to protect the loopback interface. (lines 1-20).

Then (line 22) we configure the spamd_sync sources macro, which will expand to the IP addresses of mx2 and mx3. We’ll use this later, to enable the spam database synchronisation between the three Mail Exchangers.

Following that (lines 28-40), we configure persistent PF tables, mentioned before. Note that the <spamd> table has to be called exactly that, as that name is hard-coded into spamd. Other tables to note are <bgp_spamd_bypass> and <bgp_spamd> whose names must match respective config statements for bgpd (more about that later) and <spamd-white> whose name must match the respective config in the spamd configuration file (again – more about that later).

After that, we’re scrubbing all incoming traffic of weird characteristics (e.g. drop all packets with illegal TCP flag combinations like SYN and RST; you can read more on the scrub feature in pf.conf(5) as well as in the excellent tutorial published on the OpenBSD PF FAQ page). In addition we drop traffic from sources to which our default route is not pointing via the interface such traffic came in on (this functionality is called uRFP or Unicast Reverse Path Forwarding) and also all traffic from sources which managed to get themselves added to the <bruteforce> table. (lines 42-48).

Following that there’s a pair of pass in statements (lines 51, 52) which control the inbound ICMP traffic, and finally we get to the actual email related statements.

The first of those is the pass statement in line 55, allowing traffic from the other two MX servers to port spamd-sync (udp/8025). All three servers have their spamd configured to listen on this port, and to send the database updates to the other two servers. This pretty much works out of the box.

In lines 59-62, we have a multi-line statement that allows inbound traffic in to Dovecot, to ports for imap, imaps, pop3 and pop3s (tcp 143, 993, 110, 995 respectively) and . Note that we’re only allowing packets with TCP SYN and TCP SYN/ACK flags set (i.e. only packets that properly establish the TCP session). Once the TCP session is established, the modulate state statement “kicks in” and lets the remaining packets from the current flow in as well. Finally, the last part of this statement controls thresholds for number of sessions as well as the rate at which sessions are established from the same source IP, and when those are exceeded that IP is added to the <bruteforce> table and all other states matching this source IP are killed as well, as we now think this IP is trying to compromise our server. Remember – this is where the earlier rule which drops traffic from sources listed in the <bruteforce> comes into play.

Following that, we have a set of four rules, similar to each other, on lines 66-82. These control access to ports smtp, smtps and submission (tcp 25, 465, 587). The main difference between them is whether the source IP is listed in:

• the <bgp_spamd_bypass>, <nospamd> or <spamd-white> tables, in which case the traffic is simply allowed to connect to OpenSMTPd listening on the ports above, and which is handled by the first of the four statements
• or in the <bgp_spamd> or <spamd> tables, in which case the traffic will be redirected to the spamd listening on port tcp/8025, which is handled by the second statement using the rdr-to 127.0.0.1 port spamd command
• or the source IP is not listed in any table and its destination is port tcp/25 which means this is an email that’s being sent to a user on this server for delivery, in which case the third statement kicks in and redirects the traffic to spamd by default
• or finally the source IP is not listed in any table but the traffic is sent to destination port tcp/465 or tcp/587, which means that this is someone trying to send an email through this server for delivery to a remote recipient, in which case we’re allowing the traffic to reach the smtpd directly, but we’re adding the bruteforce protection similar to the earlier statement.

Please note, the first line above are also logging to the pflog1 interface (using the pass in quick log (to pflog1) ... syntax). This is important as it’s used by spamlogd daemon, which is one of the spamd components, and which I will describe a little later.

Finally we also add an explicit statement for traffic going out to a remote destination to port tcp/25, logging it to the pflog1 interface (line 86) – again, this is used by the spamlogd to whitelist remote servers we’re making contact with, to avoid them being greylisted when the recipient responds to the email sent from us.

The remaining lines are used to explicitly block all other inbound traffic (line 90) as well as permit all outbound traffic. Note the modulate state and keep state statements at the end of those final four lines. As PF is a stateful firewall, these statements cause the PF to create a state entry for each traffic flow initiated from the server, so that the return traffic can pass through.

OpenSMTPD Configuration

The next major piece of the puzzle is the smtpd configuration, stored in /etc/mail/smtpd.conf. This one did take me a while to crack as it’s much different to what I was used to running sendmail before, but now that I have gone through the learning curve, I really appreciate the simplicity, beauty and the power of the OpenSMTPD.

# Enable queue compression and encryption
# (create key with 'openssl rand -hex 16')
queue compression
queue encryption key 24443e7a4056df59c76f9235c2360cf3 
 
# Define tables for:
# - aliases
# - domains 
# - users 
# - blacklist recipients
table aliases db:/etc/mail/aliases.db
table domains file:/etc/mail/domains
table users file:/etc/mail/users
table blacklist-recipients file:/etc/mail/blacklist-recipients
table other-relays file:/etc/mail/other-relays
 
# Define SSL certificates for host names
pki smtp.example.com certificate "/etc/ssl/smtp.example.com.crt"
pki smtp.example.com key "/etc/ssl/private/smtp.example.com.key"
 
pki mail.example.com certificate "/etc/ssl/mail.example.com.crt"
pki mail.example.com key "/etc/ssl/private/mail.example.com.key"
 
pki mx.example.com certificate "/etc/ssl/mx.example.com.crt"
pki mx.example.com key "/etc/ssl/private/mx.example.com.key"
 
max-message-size 50M
 
#
# INBOUND EMAIL CONFIGURATION
#
 
# Listen on SMTP, SMTPS, SUBMISSION in on egress interface
listen on egress tls auth-optional          # MTA
listen on egress smtps auth             # MSA (auth required)
listen on egress port submission tls-require auth   # MSA (auth required)
 
# Listen in on loopback interface 
# This is where spampd will relay email to after parsing
# - port 10026
# - tag SPAM_IN 
# - server name: smtp.example.com
listen on lo0 port 10026 tag SPAM_IN 
 
# Accept mail sent from local server to a local account
accept from local \
    for local \
    deliver to lmtp "/var/dovecot/lmtp"
 
# tagged and returned from spampd, deliver to mbox
accept tagged SPAM_IN \
    for domain <domains> virtual <users> \
    deliver to lmtp "/var/dovecot/lmtp"
 
accept tagged SPAM_IN \
    for local alias <aliases> \
    deliver to lmtp "/var/dovecot/lmtp"
 
#
# (INBOUND) START HERE
# 
 
# untagged mail is sent to SpamAssassin
accept from any \
    recipient ! <blacklist-recipients> \
    for domain <domains> \
    relay via smtp://127.0.0.1:10025    # send to spampd
 
#
# OUTBOUND EMAIL CONFIGURATION
# 
 
# Listen on SMTP on loopback interface
# - server name: smtp.example.com
 
listen on lo0
 
# Listen in on loopback interface
# - port 10028
# - tag DKIM_OUT
# - server name: smtp.example.com
 
listen on lo0 port 10028 tag DKIM_OUT
 
# Accept mail tagged DKIM-OUT (from dkimproxy_out)
# - for any 
# - relay
# - server hostname smtp.example.com
 
accept tagged DKIM_OUT \
    for any \
    relay \
    hostname smtp.example.com
 
#
# (OUTBOUND) START HERE
# 
 
# Accept mail from local
# - for any 
# - relay via smtp://127.0.0.1:10027
#   (send to dkimproxy_out)
 
accept from local \
    for any \
    relay via smtp://127.0.0.1:10027

First, in lines 1-4, I’ve enabled both the queue compression as well as encryption. This means while email files are being stored in the spool directory waiting to be processed and delivered, they are compressed to save on disk space, and then encrypted to ensure privacy. This is a completely transparent process provided to you by OpenSMTPD out of the box. At the same time, it’s still possible for the root user to see the content of the queue for debugging by using the smtpctl show queue, smtpctl show envelope and smtpctl show message commands. For more info, check out the man for smtpctl(8).

Then, in lines 6-15, we’re defining tables which we will use later in the config. These will contain the aliases database (note the backend specified for this one is db), a list of domains for which we’re accepting email (plain text file containing one domain per line), a list of users for whom we’re accepting email, a table listing recipients we don’t want to receive email for (blacklist-recipients), and a table with a list of other relays for this domain, which we’re currently not using, but which I can show you how to use.

Following is a set of statements defining the certificates and keys for SSL/TLS (lines 17-25). In this example, the primary MX server has three different hostnames: mx1.example.com, smtp.example.com and mail.example.com. OpenSMTPD supports SNI (Server Name Indication) which is a TLS extension allowing the client to specify the hostname to which it’s trying to connect during the TLS negotiation, and which allows the server to present the correct certificate (which means, it’s possible to have multiple SSL certificates on the same IP address and TCP port).

Then in line 27 we configure the maximum message size – any email larger than that will be bounced.

The following section, marked INBOUND EMAIL CONFIGURATION is where incoming email, sent to local users, is processed. I have specified three listen statements. The first one is meant to be used for accepting email coming from remote senders. It will cause OpenSMTPD to bind to port tcp/25 to an egress interface, which is another way of saying the interface via which our default gateway is reachable. This rule also specifies that TLS is to be supported (although it’s optional), and that optionally we also support user authentication.

The second listen statement binds to port tcp/465 and enables SSL. It also requires the user to authenticate before it accepts any other commands.

Finally the last listen statement binds to port tcp/587, specifies that TLS is required and that user authentication is also required before any other command is accepted.

For the next rules, it is important to note, that in the smtpd.conf the first matching rule is going to kick in, and none of the following rules will be evaluated – you need to remember this when setting up rules.

Because of that, I’ve put more specific rules first, and less specific rules to follow. To properly read this config, you need to start at lines 59-61 (marked (INBOUND) START HERE) and read the rule immediately below (line 64). In that rule, we’re accepting email sent in from anywhere, whose recipient is not in the <blacklist-recipients> table if the domain which the email was sent to is listed in the <domains> table. If those conditions are met, we then send this email to spampd, which is the daemon communicating with SpamAssassin, and which has been configured to listen on the loopback interface, on port tcp/10025. This is done using the relay via smtp://127.0.0.1:10025 statement and is possible as spampd can actually talk SMTP.

The reason why this is the least specific rule is that the traffic is not tagged with any tag. All other traffic is tagged, depending on where it comes from (as we’ll see below).

In the next step, SpamAssassin filters the email and passes it back to spampd which relays it back to OpenSMTPD, this time to 127.0.0.1:10026. And here’s where the next rule kicks in (line 43). The listen on lo0 port 10026 tag SPAM_IN rule does exactly that – every email that arrives on port tcp/10026 on lo0 will be tagged with the SPAM_IN tag. We can then match on tags in subsequent rules, as we do in rules starting on lines 51 and 55.

The first of those two rules, matches email tagged with SPAM_IN, which is sent to one of the domains from <domains> table and whose recipient is listed in the <users> table. All emails matching this rule will be delivered to the users inbox using LMTP, specifically Dovecot’s lmtp daemon.

If the email didn’t match the first rule, the second one is evaluated. This rule is very similar to the one above, except it checks if the email was sent to a recipient listed in the local <aliases> table – if it matches, the email will be delivered to inbox via LMTP.

Finally, one special case is covered by the rule at line 46, which is to accept email that was originated locally, and whose recipient is also a local user. Such email will also be delivered to inbox using LMTP. This one took me a while to figure out (although now that I have it in there it makes a lot of sense). I was used to sendmail delivering email to local accounts by default – in OpenSMTPD you need to specify all rules explicitly, which is in line with the whole philosophy of the OS. It is also worth mentioning here, that any authenticated user will be considered local by OpenSMTPD, and therefore will match rules which accept from local.

The next section of the config, marked OUTBOUND EMAIL CONFIGURATION deals with email submitted by local users for delivery to remote recipients. Similarly as above, I’ve put the rules with the least specific at the bottom, and more specific at the top of this section.

We start with line 76, which binds OpenSMTPD to the lo0 interface (which by default listens on port tcp.25).

Then we look at the bottom of the config (lines 95-106), where there’s a rule that accepts all traffic from local, with the destination set to anywhere and which relays via SMTP to 127.0.0.1, port tcp/10027. This is where the dkimproxy_out is configured to listen to, so that all outbound email has a proper DKIM signature inserted into headers. As this is the least specific rule for outbound email, it will get matched first.

Then in line 83, we also bind OpenSMTPD to interface and port lo0, tcp/10028 and tag all traffic that comes in through this interface/port with DKIM_OUT.

And finally, the rule on line 90 matches on all email that is tagged with DKIM_OUT, with destination anywhere, and relays it out setting the source hostname to smtp.example.com.

Spamd configuration

spamd.conf

The main spamd configuration file is /etc/mail/spamd.conf. This config file is used by spamd-setup(8) which is a script that should run periodically on your system and which maintains the spamd database. On my system, this is what it looks like:

# $OpenBSD: spamd.conf,v 1.4 2012/05/14 16:58:46 beck Exp $
#
# spamd(8) configuration file, read by spamd-setup(8).
# See also spamd.conf(5).
#
# Configures lists for spamd(8).
#
# Strings follow getcap(3) convention escapes, other than you
# can have a bare colon (:) inside a quoted string and it
# will deal with it. See spamd-setup(8) for more details.
#
# "all" must be here, and defines the order in which lists are applied.
# Lists specified with the :white: capability apply to the previous
# list with a :black: capability.
#
# As of November 2004, a place to search for blacklists is
#     http://spamlinks.net/filter-bl.htm
 
all:\
    :bgp-spamd:spamd_whitelist:spamd_blacklist:
 
bgp-spamd:\
     :black:\
     :msg="Your address %A has sent mail to a spamtrap\n\
      within the last 24 hours":\
     :method=file:\
     :file=/etc/mail/bgp-blacklist.txt:
 
spamd_whitelist:\
    :white:\
    :method=file:\
    :file=/etc/mail/spamd-whitelist.txt:
 
spamd_blacklist:\
    :black:\
    :msg="Your address %A is in the manual blacklist.\n\
    Unless you know the user personally and know how to contact him\n\
    via other means, you're out of luck":\
    :method=file:\
    :file=/etc/mail/spamd-blacklist.txt:

The first line, starting with the keyword all sets the order in which lists are evaluated. Please note, the order of the list is important, as where a blacklist is followed by a whitelist, if both contain an IP address, it will be removed from the final spamd whitelist. In an example above, if all three lists contained the same IP address, it would first be blacklisted (by bgp-spamd), then removed from the blacklist, but then added back in again.

As I’m using BGP for instant blacklist and whitelist propagation, I have removed the uatraps and nixspam lists out of the default configuration file, as I believe the BGP feeds contain the same IP addresses.

I have also added a bgp-spamd section as well as two entries for manual blacklist and manual whitelist (called spamd_blacklist and spamd_whitelist, respectively).

Please see spamd.conf(5) for more details.

rc configuration

The spamd itself is configured using the runtime arguments. On my systems, I had to run the following to set it up:

mx1:~$ sudo rcctl enable spamd
mx1:~$ sudo rcctl set spamd flags -v -G 2:4:864 -Y 198.51.100.2 \
> -Y 203.0.113.2 -y 192.0.2.2 -K /etc/ssl/private/smtp.example.com.key \
> -C /etc/ssl/smtp.example.com.crt
mx1:~$ sudo rcctl start spamd

The flags above configure spamd to be verbose, modify greylisting parameters (so pass time is set to 2 minutes, grey list expiry for an IP address is set to 4 hours and white list expiry is set to 864 hours), make spamd listen for database sync updates on an interface which has IP address 192.0.2.2 (our egress interface), makes spamd send database sync updates to the other two MX servers, and configures the SSL certificate and key, so spamd can support SSL/TLS.

While the above starts spamd and initially configures it, we need to add a few more extra bits to make it work properly.

spamlogd configuration

The other daemon that we need to enable is spamlogd, which is used to sniff the traffic on a pflog interface and update the spamd database accordingly. Please note, the spamd uses spamlogd to update its WHITELIST database only – therefore we only need to log the legitimate traffic to pflog1.

For this purpose, and to log the SMTP related traffic on a separate interface than the normal PF rules violations, I’ve configured the pflog1 interface on my system. You might remember, in the pf.conf file above we configured one SMTP rule to log to this interface – exactly for this reason.

Start by created the /etc/hostname.pflog1 file, containing:

up
description "spamlogd logging interface"

This will make sure that each time the system is rebooted, the pflog1 interface will be brought up. We need to also manually bring that interface up this time, so we can test that everything is working:

mx1:~$ sudo ifconfig pflog1 up description "spamlogd logging interface"

And then configure and enable the spamlogd as follows:

mx1:~$ sudo rcctl enable spamlogd
mx1:~$ sudo rcctl set spamlogd flags -l pflog1
mx1:~$ sudo rcctl start spamlogd

This will start spamlogd which will listen on interface pflog1 and update the spamd database accordingly. There’s two more bits left to configure to complete the spamd setup – OpenBGPD and spamd-setup.

OpenBGPD setup

BGP is a routing protocol used by Internet Service Providers. It’s highly scalable, able to keep millions of records in the BGP tables, and it’s what Internet routers use to build their routing tables. It’s also very flexible and can be used for a number of clever things, one of which is what the bgp-spamd project, run by Peter Hessler, is doing – propagating IP addresses which are a confirmed source of spam (blacklist), as well as a list of IP addresses which are a confirmed source of clean email (whitelist). The beauty of BGP versus other means of distributing the black and white lists is that after initial increased load while the BGP session is being set up and the BGP tables are exchanged, further updates generate very little load and are instantaneous – therefore our blacklist and whitelist can be updated pretty much in real time!

The support for BGP protocol is implemented using OpenBGPD‘s daemon – bgpd.

Following is the content of my /etc/bgpd.conf:

# $OpenBSD: bgpd.conf,v 1.1 2014/07/11 17:10:30 henning Exp $
# sample bgpd configuration file
# see bgpd.conf(5)
 
#macros
spam_rs1="64.142.121.62"    # rs.bgp-spamd.net
spam_rs2="217.31.80.170"    # eu.bgp-spamd.net
spamASN="65066"
 
AS 65500
fib-update no               # mandatory, to not update 
                            # the local routing table
 
group "spam-bgp" {
    remote-as $spamASN
    multihop 64
    announce none           # Do not send any route updates
    neighbor $spam_rs1
    neighbor $spam_rs2
}
 
# 'match' is required, to remove entries when routes are withdrawn
match from group "spam-bgp" community $spamASN:42 set pftable "bgp_spamd_bypass"
match from group "spam-bgp" community $spamASN:666 set pftable "bgp_spamd"

It’s a really simple config, with only one “magic” bit.

We start with defining three macros: two for the OpenBGPD Team’s route servers (spam_rs1 and spam_rs2) which is where we’re going to establish the BGP sessions to, and the third one is just a label for the OpenBGPD’s Autonomous System Number (spamASN).

AS Numbers are what BGP uses to group IP prefixes into – typically, an Internet Service Provider will have a number of IPv4 and IPv6 ranges (called prefixes), and a single AS Number which all these IPv4/IPv6 addresses are grouped within. BGP routers then calculate a path to the destination IP by looking at which AS Numbers the prefix is visible behind – this is called AS-Path.

We then need to define our own AS Number – I’ve set mine to 65500, which is in the private AS Number range (65413 – 65535). Unless your organisation has been assigned a public AS Number, you should too.

We then have an important and mandatory option for this config, which is to disable updating the system’s routing table with the information in the BGP table (fib-update no). This is important as even though we’re using what has been designed as a routing protocol, we’re using it for a whole different reason and therefore we don’t want the BGP to modify your server’s routing table.

Then in lines 14-21, we configure the actual BGP sessions to the two Route Servers, making sure we’re not advertising any routes back to them.

Finally, the last two lines, are where the magic happens. Line 24 matches all prefixes which come from the above Route Servers, and which have their BGP Community (which is an extended attribute you can think of somewhat like a “colour”) set to 65066:42 and adds them to the PF's <bgp_spamd_bypass> table. Similarly, the last line (25) does a similar thing for prefixes marked with BGP Community 65066:666 and adds them to the <bgp_spamd> table. Again – if you remember, we have configured those two tables in /etc/pf.conf above as persistent tables, and use them to decide where to send the traffic based on which IP address it came from (and whether the IP address was black- or whitelisted).

Any BGP table update will be propagated immediately to the PF config – kind of neat, isn’t it?

spamd-setup

Finally, we need to make sure our spamd database is kept in sync and up to date with all those changes as well. To do that, I’ve written a really short bash script, which I call from cron every 20 minutes or so, and which is saved to /usr/local/bin/bgp-spamd-update.sh:

#!/usr/local/bin/bash
#
#
 
ASN=65066
COMMUNITY=666
 
bgpctl show rib community ${ASN}:${COMMUNITY} | sed -e '1,4d' -e 's/\/.*$//' -e 's/[ \*\>]*//' > /etc/mail/bgp-blacklist.txt
 
/usr/libexec/spamd-setup 
 
/usr/bin/logger -p mail.info -t spamd-update "spamd black list updated; bgp_spamd: $(/sbin/pfctl -t bgp_spamd -T show | /usr/bin/wc -l), bgp_spamd_bypass: $(/sbin/pfctl -t bgp_spamd_bypass -T show | /usr/bin/wc -l)"

What this does is it saves the list of IP addresses in the BGP table which are marked with 65066:666 community into the /etc/mail/bgp-blacklist.txt, and then runs the /usr/libexec/spamd-setup which uses that file, as well as the other ones we configured earlier in spamd.conf to populate the spamd database.

The final line is just a message that’s logged via syslog, and which logs how many prefixes we have in the bgp_spamd and bgp_spamd_bypass tables.

There’s one more thing to configure to finish handling the incoming email: SpamAssassin.

SpamAssassin and spampd configuration

Similarly to the other daemons, first we need to enable both SpamAssassin and spampd and configure their run time arguments:

mx1:~$ sudo rcctl enable spamassassin
mx1:~$ sudo rcctl start spamassassin
mx1:~$ sudo rcctl enable spampd
mx1:~$ sudo rcctl set spampd flags "--port=10025 --relayhost=127.0.0.1:10026 --tagall --aw --rh --maxsize=256 -pid=/var/spampd/spampd.pid"
mx1:~$ sudo rcctl start spampd

We’ll just use the default settings for SpamAssassin here, as fine tuning its rules is beyond the scope of this article. I would strongly suggest you look at features like Auto Whitelists or Bayesian filter, as well as enabling some of the extra modules.

Note the configuration of spampd in line 4 above – we’re telling it to bind to port tcp/10025 and to relay messages that SpamAssassin is done with to 127.0.0.1:10026. Again – if you remember, we have a section in /etc/smtpd.conf which makes OpenSMTPD listen to port tcp/10026 on lo0 interface and tag all email coming in this way with the SPAM_IN tag – this is the other “end” of that connection.

Now that we have all bits in place for handling incoming email, there’s one more thing we need to configure for sending email out – the DKIM Proxy.

DKIM Proxy configuration

First we need to edit the configuration file, which should be at /etc/dkimproxy_out.conf:

# specify what address/port DKIMproxy should listen on
listen    127.0.0.1:10027
 
# specify what address/port DKIMproxy forwards mail to
relay     127.0.0.1:10028
 
# specify what domains DKIMproxy can sign for (comma-separated, no spaces)
domain    example.com
 
# specify what signatures to add
signature dkim(c=relaxed)
signature domainkeys(c=nofws)
 
# specify location of the private key
keyfile   /var/dkimproxy/default.private
 
# specify the selector (i.e. the name of the key record put in DNS)
selector  default
 
# control how many processes DKIMproxy uses
#  - more information on these options (and others) can be found by
#    running `perldoc Net::Server::PreFork'.
#min_servers 5
#min_spare_servers 2

As you can see from the file above, the dkimproxy_out is listening on port tcp/10027 for emails sent from OpenSMTPD, and once those are DKIM signed, it’s relaying them back to OpenSMTPD to 127.0.0.1:10028. Recall the corresponding settings in the /etc/smtpd.conf above and now you have a full picture of what’s going on there.

The next step is to generate the key which the DKIM Proxy should use to sign outgoing email. Please note, if you’re migrating from another setup and you already have a key, you may be able to use it instead.

To generate the key, run the following command:

mx1:~$ sudo dkim-genkey

This will create two files in /var/dkimproxy: default.private and default.txt.

The default.private contains the private key used for email signing, and you want to keep this key private.

The default.txt contains the DNS record you need to add to your zone configuration file for each domain you have (in the case of this document we’re only using one domain, example.com). What you do is literally copy and paste the content of this file into your zone configuration file. The DNS configuration is out of scope of this article, but a quick Google search will point you in the right direction.

Note, that both the default configuration file for dkimproxy_out, as well as dkim-genkey assume that your key selector (i.e. a label or a name) is going to be set to default – if you’d like to change it, make a suitable change in /etc/dkimproxy_out.conf (line 18, after the selector statement) as well as in the default.txt file and the DNS zone configuration file, as all of these need to match. Also, it would be a good idea to rename the two key files to reflect the selector name.

Finally, we need to enable and start the dkimproxy_out by running the following commands:

mx1:~$ sudo rcctl enable dkimproxy_out
mx1:~$ sudo rcctl start dkimproxy_out

This concludes the SMTP setup for the Primary Mail Server. The other thing we will need to do is to configure Dovecot so our users can actually access their email via IMAP or POP3.

Dovecot configuration

Dovecot is one of the most popular applications handling user email retrieval. The way it’s configured by default is pretty much spot on for my setup, although I did have to tweak a few things here and there.

Instead of pasting in complete configuration files (which there are plenty of for Dovecot, and which are rather long as they are quite verbosely commented), I will list the files and changes I made against the defaults, and provide you with a quick rationale.

/etc/dovecot/dovecot.conf

Find the protocols line and enable the protocols you want to enable. In my case these are pop3, imap and lmtp.

In addition, as I’m running IPv6 on my servers as well as IPv4, I had to change the listen statement to include :: after * so Dovecot uses both address families.

# [...]
protocols = imap pop3 lmtp
# [...]
listen = *, ::
# [...]

/etc/dovecot/conf.d/10-auth.conf

The main thing here is to disable plain text authentication. Please note, that Dovecot will still permit plain text authentication if SSL/TLS is used.

# [...]
disable_plaintext_auth = yes
# [...]

/etc/dovecot/conf.d/10-mail.conf

For historical reasons, my server is storing user email in mbox. As some of the users have decades worth of email (no, really), it’s difficult to migrate them off. Because of that, dovecot needs to be configured accordingly. Also, if you’re delivering mail to users’ mailboxes directly with smtpd (using the deliver to statement), you need to make sure that both of these match.

In addition it is worth looking at the first_valid_uid and last_valid_uid statements in this file to ensure the bad guys can’t log in as daemons or system users.

# [...]
mail_location = mbox:~/mail:INBOX=/var/mail/%u
# [...]

/etc/dovecot/conf.d/10-ssl.conf

This is where you configure the SSL settings for your services.

Apparently Dovecot supports SNI, so depending on which host name a client is connecting to, Dovecot can serve the corresponding SSL certificate, however it didn’t seem to be working for me – in the end I’ve decided to use a single certificate and a single host name and just tell all my users to use that to connect to either IMAP or POP3.

I have also disabled weak ciphers and protocols, and configured Dovecot to prefer the server cipher list so that the clients can’t choose to use a weak cipher instead.

# [...]
ssl = yes
# [...]
ssl_cert = </etc/ssl/mail.example.com.crt
ssl_key = </etc/ssl/private/mail.example.com.key
# [...]
ssl_dh_parameters_length = 4096
# [...]
ssl_protocols = !SSLv2 !SSLv3
# [...]
ssl_cipher_list = ALL:!LOW:!EXP:!aNULL
# [...]
ssl_prefer_server_ciphers = yes
# [...]

/etc/dovecot/conf.d/15-lda.conf

Set the postmaster field to postmaster@example.com and hostname to mail.example.com

Adjust login resources limit for dovecot user

As per Dovecot Readme, it is a good idea to adjust the resources limit in the system for the dovecot user, otherwise you’ll run into an issue with too many open files.

In /etc/login.conf add a new section at the bottom for the dovecot user, like so:

#
# Dovecot
#

dovecot:\
        :openfiles-cur=2048:\
        :openfiles-max=4096:\
        :tc=daemon:

This increases the limit of simultaneously open files and puts dovecot process in its own login class (by default it runs in the daemon class).

Enable dovecot

Finally, to enable and start Dovecot, type the following commands:

mx1:~$ sudo rcctl enable dovecot
mx1:~$ sudo rcctl start dovecot

There are other things you can (and probably should) do with Dovecot like using the sieve plugin to give your users ability to configure mail filters on the server (so that it’s the server that moves SPAM to the Junk folder instead of the client), or to enable disk quota support which allows for graceful handling of cases where users email fills their assigned quota. I’ll leave those to you as an exercise.

Primary MX Setup Summary

This concludes the setup of the Primary Mail server mx1.example.com. We have configured the PF to redirect email coming from unknown or blacklisted sources to spamd for greylisting or tarpitting; OpenBGPD for maintaining in real-time various black and white lists for spamd; OpenSMTPD for handling incoming and outgoing email, with SSL/TLS, user authentication and various forwarding rules to ensure inbound email is evaluated and classified by SpamAssassin while outbound email is DKIM signed; SpamAssasin and spampd for mail filtering; dkimproxy_out for outbound email signing and finally Dovecot for handling delivery of email to users mailboxes and email retrieval for our users.

This should give you a fully working and very nicely configured email server to start with. There’s more we can do, however, by setting up Backup MX servers, whose main reason to be is to queue email sent to your users during the time your primary server might be offline for maintenance, or because of network issues.

You can read about this in part two of this series. The good news is that the Backup MX servers have only a subset of the configuration of the Primary one, and therefore should be much easier and quicker to deploy. Stay tuned!