Technology

I’ve been administering e-mail servers since the early 2000s, for both my myself and for various jobs. For a brief period I stopped hosting my own e-mail, but returned to running my own stack due to the revelation of domestic spying in 2013. Even though the larger providers have made e-mail less reliable than it once was, I’m still glad I host my own e-mail. I had been using an OpenBSD 6.3 VM for e-mail, and couldn’t upgrade to OpenSMTPD 6.4+ because of some big configuration file changes. Thanks to many good 6.3 → 6.4+ tutorials, I finally tackled this lingering piece of technical debt, and migrated my e-mail from an OpenBSD VM to my standard Docker infrastructure.

Technical Debt

My OpenBSD 6.3 VM had been chugging along fine, but eventually certbot was no longer able to get new LetsEncrypt certificates. ACMEv1 is deprecated and the version of certbot in the OpenBSD 6.3 packages was too old to use ACMEv2. I attempted to manually install a newer version of certbot via pip. However, newer versions of certbot depend on cryptography which depends on Rust¹, and the version of Rust that came with OpenBSD 6.3 was too old to build cryptography. A friend recommended dehydrated, which allows for pulling LetsEncrypted certificates using Bash. Although these certificates worked with Dovecot/IMAP, OpenSMTPD 6.3 couldn’t use the newer certificate types it produced. At this point I decided, rather than fighting to keep the old server alive, it was finally time to bite the bullet and migrate.

Configuration

My original stack consisted of opensmtpd, spamassassin/spampd, clamav, dkim_proxy, dovecot and procmail. My new stack is similar, except that procmail is no longer maintained, and has been replaced by fdm. I briefly looked at rspamd (which could have replaced dkim_proxy, spamassassin and clamav), but decided instead to stick to what I know. I wanted to minimize changes and make the transition as smooth as possible. My original opensmtpd 6.3 configuration looked like the following.

pki mail.example.com key "/etc/letsencrypt/live/mail.example.com/privkey.pem"
pki mail.example.com certificate "/etc/letsencrypt/live/mail.example.com/fullchain.pem"

table vdoms  "/etc/mail/vusers"
table vusers "/etc/mail/vdomains"

listen on lo0 port 10027 tag SPAMD
listen on lo0 port 10028 tag CLAM
listen on lo0 port 10030 tag DKIM

listen on vio1 port smtp

listen on egress port smtp tls pki mail.example.com
listen on egress port 465 smtps pki mail.example.com
listen on egress port submission tls-require pki mail.example.com auth

accept tagged CLAM for domain <vdoms> virtual <vusers> deliver to mda "procmail -f -"
accept tagged SPAMD for any relay via "smtp://127.0.0.1:10026"
accept from any for domain <vdoms> relay via "smtp://127.0.0.1:10025"

accept tagged DKIM for any relay hostname mail.example.com
accept from local for any relay via smtp://127.0.0.1:10029

I used an excellent guide, written by Gilles Chehade, as my starting point for migrating my configuration file. It goes over adjusting the previous accept lines into the new match and action directives, the new syntax for listen and a host of other conversions². I ended up with a new configuration that looked like the following:

pki mail.example.com key  "/etc/letsencrypt/live/mail.example.com/privkey.pem"
pki mail.example.com cert "/etc/letsencrypt/live/mail.example.com/fullchain.pem"

table vdoms  "/mail/db/vdomains"
table vusers "/mail/db/vusers"
table addrs  "/mail/db/addrs"
table passwd passwd:/mail/db/userdb

action "deliver" mda "/usr/bin/fdm -a stdin -f /mail/config/fdm.conf fetch" virtual <vusers>
action "relay"   relay

# Outgoing Filter
action relay_dkimproxy_out relay host smtp://127.0.0.1:10029
listen on 127.0.0.1 port 10030 tag DKIM_IN

# Incoming Filters
action relay_clamav_out relay host smtp://127.0.0.1:10026
listen on 127.0.0.1 port 10028 tag CLAM_IN
action relay_spampd_out relay host smtp://127.0.0.1:10025
listen on 127.0.0.1 port 10027 tag SPAMD_IN

listen on 0.0.0.0 port smtp tls pki mail.example.com
listen on :: port smtp tls pki mail.example.com

listen on 0.0.0.0 port smtps tls-require pki mail.example.com
listen on :: port smtps tls-require pki mail.example.com

listen on 0.0.0.0 port submission tls-require pki mail.example.com hostname mail.example.com auth <passwd>
listen on :: port submission tls-require pki mail.example.com hostname mail.example.com auth <passwd>

# Incoming
match tag CLAM_IN for any action relay_spampd_out
match tag SPAMD_IN for any action "deliver"
#match from any for domain <vdoms> action relay_clamav_out
match from any for rcpt-to <addrs> action relay_clamav_out

# Outgoing
match tag DKIM_IN for any action "relay"
match auth from auth for any action relay_dkimproxy_out

Docker

If you’re starting from scratch, there are existing solutions that run in Docker. Mailu is composed of standard e-mail services such as postfix and dovecot, combined with a web interface and configuration tools. Maddy is written in Go and attempts to be a full stack replacement, implementing its own SMTP, IMAP, DKIM and other services.

I considered these tools, but decided to implement my old OpenBSD stack inside an Alpine container, to minimize the amount of changes needed. I wanted to bring over my Maildir folder without having to convert my old messages into a new system. E-mail is pretty important for most of the work I do, and I was aiming for minimal downtime. If you are just starting out, I would take a look at some of the solutions I mentioned first. My solution is a good starting point if you want fine grained control of your e-mail stack, using proven applications and services.

The State outside the Container

Typically, a single Docker container has one concern, and ideally, one running process. E-mail involves several complementary services. I could have created a container for each service. Instead, I committed the unforgivable sin of the container world and ran them all in a single container using supervisord to managed their lifecycle. I also configured most services to use the /mail folder for their state, which I mapped to an external volume. In total, I use three volumes to manage state:

• mail:/mail to hold configuration and mail
• mail_queue:/var/spool.smtpd for the opensmtpd mail queue
• mail_letsencrypt:/etc/letsencrypt for TLS certificates

Neither certbot or opensmtpd seem to allow changing their settings or storage locations, which is why I didn’t place them under /mail. The structure of /mail within the container looks like the following:

mail
├── bin
│   ├── certbot-deploy-hook
│   ├── get_certs
│   └── startup
├── config
│   ├── clamd.conf
│   ├── clamsmtpd.conf
│   ├── dkim.key
│   ├── dkimproxy_out.conf
│   ├── fdm.conf
│   ├── freshclam.conf
│   └── smtpd.conf
├── db
│   ├── addrs
│   ├── clamav
│   │   └──...
│   ├── spamassassin
│   │   └──...
│   ├── spampd
│   │   └──...
│   ├── userdb
│   ├── vdomains
│   └── vusers
├── log
│   ├── clamd.log
│   ├── clamsmtpd.log
│   ├── cron.log
│   ├── dkimproxy.log
│   ├── dovecot.log
│   ├── freshclam.log
│   ├── smtpd.log
│   └── spampd.log
└── spool
    ├── bob
    │   └── Mail
    │       └──...
    ├── sysmail
    │   └── Mail
    │       └──...
    └── vmail
        └── Mail
            └──...

There are three scripts in /mail/bin. I could have placed these within the container, but I wanted to separate out everything that referenced specific domain names and settings. The get_certs script uses LetsEncrypt to retrieve TLS certificates:

#!/bin/sh

/usr/bin/certbot certonly --standalone --preferred-challenges http \
   --http-01-port 80 --agree-tos --renew-by-default --non-interactive \
   --email notify@example.com -d mail.example.com \
   --deploy-hook /mail/bin/certbot-deploy-hook

The certbot-deploy-hook script is run after a certificate renewal, to tell other services to load the new certs:

#!/bin/sh
chmod 0600 $RENEWED_LINEAGE/privkey.pem
echo "Send HUP to smtpd and dovecot"
supervisorctl signal HUP opensmtpd
supervisorctl signal HUP dovecot

The final script is startup which is called from the container startup script, in order to create the correct symbolic links:

 cat bin/startup
#!/bin/sh

ln -sf /etc/letsencrypt/live/mail.example.com/privkey.pem /etc/ssl/dovecot/server.key
ln -sf /etc/letsencrypt/live/mail.example.com/fullchain.pem /etc/ssl/dovecot/server.pem
ln -sf /mail/db/userdb /etc/dovecot/users

Within the Container

The following is the Alpine based Dockerfile I used for my e-mail container. The lineinfile script is just a bash implementation I found of the similar Ansible function³. I also download a release of spampd. There is a version within the Alpine testing repository, but it’s an older version that will break unless a syslog daemon is running. The vmail user is used for mailbox and delivery permissions. It’s important to run this with the container’s hostname set to the DNS name of your mail server. That was the only way I could get opensmtpd to reliable announce the correct hostname banner. If using the docker cli, you can use --hostname, or the Hostname parameter in the Docker Engine API.

FROM alpine:3.13.6

RUN apk update && \
    apk add opensmtpd supervisor certbot dkimproxy clamav-libunrar spamassassin \
    dovecot-pgsql dovecot perl-mail-spamassassin freshclam clamsmtp clamav-daemon opensmtpd-table-passwd && \
    apk add fdm --repository=http://dl-cdn.alpinelinux.org/alpine/edge/testing


# SpamPD in alpine repo is out of date
RUN wget https://github.com/mpaperno/spampd/archive/refs/tags/2.61.tar.gz
RUN echo "91e60f10745ea4f9c27b9e57619a1bf246ab9a88ea1b88c4f39f8af607e2dbae  2.61.tar.gz" | sha256sum -c
RUN tar xvfz 2.61.tar.gz
RUN rm 2.61.tar.gz

COPY spamassassin-local.cf /etc/mail/spamassassin/local.cf

COPY supervisor.conf /etc/supervisord.conf
COPY crontab /var/spool/cron/crontabs/root

COPY lineinfile /usr/share/misc/lineinfile

RUN adduser -h /mail/spool -s /bin/false -D -u 2000 -g 2000 vmail

VOLUME ["/mail"]

STOPSIGNAL SIGTERM

COPY startup /
RUN chmod 755 /startup

CMD ["/startup"]

The spamassassin-local.cf configuration is very simple, adding a string to the subject that can be filtered via fdm later.

rewrite_header Subject ***Spam***

The crontab is fairly straight forward too. It’s used to update certificates, anti-virus and spamassassin.

1 1 * * 1 /mail/bin/get_certs
1 */12 * * * /usr/bin/freshclam --foreground --config=/mail/config/freshclam.conf --daemon-notify=/mail/config/clamd.conf
30 */12 * * * sa-update -v --updatedir /mail/db/spamassassin

The startup script chains the startup created in the mail volume, does some configuration, and then starts supervisord.

#!/bin/sh

if [ -x /mail/bin/startup ]; then
  echo "Running /mail/bin/startup"
  /mail/bin/startup
fi

echo "Setting up spampd"
mkdir -p /mail/db/spampd || true
chown vmail:vmail /mail/db/spampd

echo "Setting up Dovecot"
source /usr/share/misc/lineinfile
lineinfile '^#?mail_location' "mail_location = maildir:/mail/spool/%n/Mail" /etc/dovecot/conf.d/10-mail.conf
lineinfile "^#?protocols =" "protocols = imap" /etc/dovecot/dovecot.conf
lineinfile "^#log_path =" "log_path = /dev/stderr" /etc/dovecot/conf.d/10-logging.conf

echo "DKIM Proxy Permissions"
chown dkimproxy:dkimproxy /mail/config/dkim.key /mail/config/dkimproxy_out.conf

exec /usr/bin/supervisord -n -c /etc/supervisord.conf

Finally, supervisord starts all the individual services required for a full e-mail stack. Some services can maintain their own logs and log rotation. For the ones that don’t, supervisord can route standard out and error to files and handle log rotation.

[unix_http_server]
file=/run/supervisor.sock   ; (the path to the socket file)

[supervisord]
logfile=/mail/log/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10)
loglevel=info                ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false               ; (start in foreground if true;default false)
minfds=1024                  ; (min. avail startup file descriptors;default 1024)
minprocs=200                 ; (min. avail process descriptors;default 200)
user=root        ;

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///run/supervisor.sock ; use a unix:// URL  for a unix socket

[program:opensmtpd]
command = /usr/sbin/smtpd -d -f /mail/config/smtpd.conf
autostart=true
autorestart=true
priority=5
stdout_logfile=/mail/log/smtpd.log
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=10
redirect_stderr=true

[program:cron]
command = crond -f -d 8
autostart=true
autorestart=true
priority=5
stdout_logfile=/mail/log/cron.log
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=10
redirect_stderr=true

[program:dovecot]
command = dovecot -F -c /etc/dovecot/dovecot.conf
autostart=true
autorestart=true
priority=5
stdout_logfile=/mail/log/dovecot.log
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=10
redirect_stderr=true

[program:dkimproxy]
command =/usr/sbin/dkimproxy.out --conf_file=/mail/config/dkimproxy_out.conf --user=dkimproxy
autostart=true
autorestart=true
priority=5
stdout_logfile=/mail/log/dkimproxy.log
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=10
redirect_stderr=true

[program:clamsmtpd]
command = /usr/sbin/clamsmtpd -f /mail/config/clamsmtpd.conf -d 1
autostart=true
autorestart=true
priority=5
stdout_logfile=/mail/log/clamsmtpd.log
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=10
redirect_stderr=true

[program:clamd]
command = /usr/sbin/clamd --config-file=/mail/config/clamd.conf --foreground
autostart=true
autorestart=true
priority=5

[program:spampd]
command = /spampd-2.61/spampd.pl --port=10025 --relayhost=127.0.0.1:10027 --tagall --nodetach --homedir=/mail/db/spampd --logfile /mail/log/spampd.log --saconfig=/mail/db/spamassassin/updates_spamassassin_org.cf -u vmail -g vmail
autostart=true
autorestart=true
priority=5

The full container can be found in the bee2 project, which I use to manage my self-hosted services.

The Stages of E-mail

The basic flow of the e-mail is the same from my original to my new configuration. Incoming mail takes the following route:

• Match valid rcpt-to in addrs table
• Relay mail to ClamAV filter on 10026
• ClamAV returns mail on 10028, which is tagged as CLAM_IN
• Relay CLAM_IN to SpamPD on 10025
• SpamPD preforms Spamassassin checks and adds ***SPAM*** to the subject fields of SPAM
• SpamPD returns mail on 10027, which is tagged as SPAMD_IN
• Relay SPAMD_IN to deliver action
• Deliver action sends e-mail via stdin to fdm
• fdm uses /mail/config/fdm.conf to match rules and forward deliver to dovecot

Outgoing e-mail comes in on the submission port and gets relayed via dkimproxy_out in order to be signed with a DKIM key.

/mail/config/clamd.conf

The ClamAV daemon doesn’t scan e-mails directly. It must be running so ClamSMTPD can use it for scanning messages via the clamd.sock

LogFile /mail/log/clamd.log
DatabaseDirectory /mail/db/clamav
LogFileMaxSize 2M
LogTime yes
LogRotate yes
LocalSocket /tmp/clamd.sock
User clamav
MaxRecursion 12

/mail/config/clamsmtpd.conf

ClamSMTPD receives e-mail from and sends it back to OpenSMTPD, using the clamd.sock to communicate with ClamAV to scan messages for viruses. Messages with viruses are quarantined and not delivered back to OpenSMTPD.

OutAddress: 10028
Listen: 0.0.0.0:10026
ClamAddress: /tmp/clamd.sock
Header: X-Virus-Scanned: ClamAV using ClamSMTP
TempDirectory: /tmp
Action: drop
Quarantine: on
TransparentProxy: off
User: clamav

/mail/config/freshclam.conf

Freshclam is run periodically via the crontab listed above to refresh the antivirus rules.

UpdateLogFile /mail/log/freshclam.log
DatabaseDirectory /mail/db/clamav
DatabaseOwner clamav
LogFileMaxSize 2M
LogTime yes
Foreground yes
DatabaseMirror db.us.ipv6.clamav.net
DatabaseMirror database.clamav.net

/mail/config/dkimproxy_out.conf

The dkim.key must be generated and the public key added to the domain’s DNS records. See the dkimproxy documentation for more information⁴.

listen    127.0.0.1:10029
relay     127.0.0.1:10030
domain    example.com,example.net
signature dkim(c=relaxed)
signature domainkeys(c=nofws)
keyfile   /mail/config/dkim.key
selector  dkim1

Migrating from procmail to fdm

Procmail was one of the components I had to replace, as it’s no longer maintained and cannot be found in most package repositories⁵. The following is an example of what my previous .procmailrc looked like. It’s designed to accept e-mail from multiple addresses and place them in respective folders of a single user. The default e-mail address isn’t specified as it is the default delivery in the last rule.

LOGFILE="$HOME/procmail.log"
MAILDIR="$HOME/Mail/"
DEFAULT="$HOME/Mail/"
DELIVER="/usr/local/libexec/dovecot/deliver"

:0 w
* ^X-Spam-Status: Yes
| $DELIVER -m Spam

:0 w
* ^(To|Cc|Received):.+notifications@example\.com
| $DELIVER -m Notices

:0 w
* ^(To|Cc|Received):.+mailinglists@example\.com
| $DELIVER -m Mailing\ Lists

:0 w
* ^(To|Cc|Received):.+system@example\.com
| $DELIVER -m System

:0 w
* ^(To|Cc|Received):.+dmarc@example\.com
| $DELIVER -m DMARC

:0 w
| $DELIVER

Converting this .procmailrc looks like the following. As you can tell from the original and new smtpd.conf, instead of using an individual/per-user configuration file as I did with procmail, I have one global fdm.conf which all e-mail is relayed through. This works since I’m running a single user e-mail instance, but would have to be redesigned if you ran an e-mail system with multiple users. With this configuration, the default INBOX route must be defined (e.g. bob@example.com). Before there would have been a separate .procmailrc for the sysmail mail user (which handles e-mails for the second domain, example.net), but here those rules are combined into one fdm.conf.

#/mail/config/fdm.conf

$deliver = '/usr/libexec/dovecot/deliver'

account "stdin" disabled stdin

action "bob_spam" pipe "${deliver} -d bob -m Spam"
action "bob_inbox" pipe "${deliver} -d bob -m INBOX"
action "bob_notifications" pipe "${deliver} -d bob -m Notifications"
action "bob_mailinglists" pipe "${deliver} -d bob -m Mailing\ Lists"
action "bob_system" pipe "${deliver} -d bob -m System"
action "bob_dmarc" pipe "${deliver} -d bob -m DMARC"

action "network" pipe "${deliver} -d sysmail -m Network"

match "^X-Spam-Status: Yes" in headers action "bob_spam"
match "^(To|Cc|Received):.+bob@example\\.com" in headers action "bob_inbox"
match "^(To|Cc|Received):.+notifications@example\\.com" in headers action "bob_notices"
match "^(To|Cc|Received):.+mailinglists@example\\.com" in headers action "bob_lists"
match "^(To|Cc|Received):.+system@example\\.com" in headers action "bob_system"
match "^(To|Cc|Received):.+dmarc@example\\.com" in headers action "bob_dmarc"

match "^(To|Cc|Received):.+network@example\\.net" in headers action "network"

Tables

The tables don’t change much between OpenSMTPD 6.3 to 6.4+. However, in migrating to a Docker container, I’m no longer using system users or PAM authentication. I wanted to minimize replicating data, so I used the opensmtpd-table-passwd package so OpenSMTPD could read from the same passwd table used by Dovecot. As you can see from the Dockerfile above, all mail data is managed by the user vmail, with individual users handled by OpenSMTPD and Dovecot respectively. Starting with the users, the passwd table located in /mail/db/userdb looks like the following:

bob:$6$xx<some long encrypted password string>:vmail:2000:2000:/mail/spool/bob::userdb_mail=maildir:/mail/spool/bob/Mail
sysmail:$6$xx<some long encrypted password string>:vmail:2000:2000:/mail/spool/sysmail::userdb_mail=maildir:/mail/spool/sysmail/Mail

The /mail/db/vdomains is simply a list of domains we accept mail from.

example.com
example.net

The /mail/db/vusers simply maps all valid e-mail addresses to the single vmail user:

bob@example.com vmail
notifications@example.com vmail
mailinglists@example.com vmail
system@example.com vmail
dmarc@example.com vmail
networking@example.net vmail

The /mail/db/addrs file is the exact same as the vusers file, except without the username mapping. This duplication is required due to the matching rule match from any for rcpt-to <addrs> action relay_clamav_out within the smtpd.conf. This table isn’t strictly necessary. You can change the match rule to be match from any for domain <vdoms> action relay_clamav_out and avoid the use of this table. However, checking against just the domain will result in OpenSMTPD accepting e-mails for invalid addresses. Without the use of rcpt-to, there is no way for the mail server to resolve if an e-mail is valid until after it’s accepted into the pipeline⁶. Using rcpt-to combined with this tables prevents backscatter bounces.

bob@example.com
notifications@example.com
mailinglists@example.com
system@example.com
dmarc@example.com
networking@example.net

There is a lot of duplicated information between some of these tables. Both Dovecot and OpenSMTPD support sqlite adapters for queering this information, and the entire four file structure could potentially be simplified to a single SQL file with the correct queries. I might attempt this at a later time, but this works for now.

DNS

The DNS is pretty straight forward. There are A and AAAA records for the mail server itself. For every domain we handle mail for, they will need an MX record pointing to the mail server’s DNS, as well as TXT records for SPF, DKIM signatures and DMARC. The DKIM record will need to contain the public key generated for DKIM proxy. In the following set of tables, example.net handles none of its own e-mail, delegating everything to example.com via the MX record. Per the configuration for DKIM proxy, e-mails from both domains get signed by the same keys.

example.com

example.net

Verification

There are a variety of websites that you can use to test your SMTP server to ensure it’s not an open relay and that your SSL certificates are valid. I’ve found that one of the best services is mail-tester.com. It presents you with a randomly generated address to send a message to, and then proceeds to give you an analysis of the entire message and handling pipeline. Using this service, I discovered a mismatched between my hostname and EHLO banner, as well as potential IP blacklist issues.

Closing Thoughts

Setting up e-mail is not easy. Newer protocols are often designed entirely around HTTPS or web sockets, and they’re typically implemented with one application that both sends and receives messages. If you’re setting up an e-mail server from scratch, have no mail to import/migrate, and want to use Docker, I’d suggest looking at one of the solutions I mentioned in the Docker section of this post. If you’re in my situation where you’re trying to migrate an old e-mail stack and have the time to switch things out, I’d recommend looking at Rspamd to replace the services I used for spam, antivirus and dkim. You can also minimize the number of OpenSMTPD tables by looking into sqlite queries for tables. I may attempt to implement some of these suggestions in the future.

I realize few people would look to my setup as something to emulate. This post was more to document some of the migration challenges I faced upgrading from OpenSMTPD 6.3, as well as examples for migrating from procmail to fdm. I hope it may help some people seeking to make those transitions, as well as displaying the complexity of modern e-mail stacks.

E-mail is an ancient protocol. It’s one of the oldest federated messaging protocols on the Internet that’s still in wide spread use (NNTP could be considered another, but it is no where near as well known today). Where newer services use TXT and SRV records in DNS to indicate where to locate servers, e-mail has its own specific MX records. Where newer protocols have one service to handle all communication, e-mail uses different services and ports for sending and receiving. Where other protocol attempt to secure traffic by domains and TLS, e-mail has bolted on features such as SPF record and DKIM signatures.

Despite all the cruft, e-mail is still important, if not vital, to handling account credentials, password resets, alert messages, work correspondence and even personal messages. It’s not very secure compared to newer protocols, which offer much simpler end-to-end encryption and key exchange. Yet for better or for worse, e-mail isn’t going away any time soon. Running an e-mail server is challenging, but if you can host your own e-mail, running anything else becomes trivial.