Technology

Recently I added e-mail support to Bee2, a tool I use to provision servers and services for personal projects. My existing e-mail server ran on an openSUSE 13.2 box which stopped receiving security updates in January of 2017. I’ve been building Ansible roles to provision a replacement running on OpenBSD, and I found an interesting bug with a proxy filtering service called SpamPD. The bug prevents SpamPD from starting correctly with the standard OpenBSD rc.d service scripts due to the weird ways the OpenBSD init process works.

The best practice for starting a service in OpenBSD is to enable the service in /etc/rc.conf.local, overriding defaults in /etc/rc.conf and possibly adding startup flags to the daemon’s processes. So to setup SpamPD to fit into my mail pipeline, I’d need something like the following:

pkg_scripts=freshclam clamd clamsmtpd dkimproxy_out dovecot spampd
spampd_flags="--port=10025 --relayhost=127.0.0.1:10027 --tagall -pid=/var/spampd/spampd.pid -aw"

In the above rc.conf.local I’m starting several mail services like clamsmtpd for anti-virus and dkimproxy_out for adding DKIM signatures to outbound e-mail. spampd is listed along with additional startup flags. SpamPD doesn’t use a configuration file like the other services, so it must be configured using startup flags.

The trouble is, if I start an OpenBSD box with this configuration, SpamPD will come up without any of the flags I placed in rc.conf.local set:

$ ps auxwww | grep spampd
_spampd  79539  6.6  8.4 74884 86748 C0- S      5:36AM    0:02.28 /usr/bin/perl -T /usr/local/sbin/spampd
_spampd  82812  0.0  0.5 74884  4816 ??  S      5:36AM    0:00.00 /usr/bin/perl -T /usr/local/sbin/spampd
_spampd  99047  0.0  0.7 74884  6788 ??  S      5:36AM    0:00.05 /usr/bin/perl -T /usr/local/sbin/spampd
_spampd  92986  0.0  0.5 74884  4964 ??  S      5:36AM    0:00.00 /usr/bin/perl -T /usr/local/sbin/spampd
_spampd  10085  0.0  0.5 74884  5068 ??  S      5:36AM    0:00.05 /usr/bin/perl -T /usr/local/sbin/spampd
_spampd  89447  0.0  0.5 74884  5076 ??  S      5:36AM    0:00.07 /usr/bin/perl -T /usr/local/sbin/spampd

For the longest time, I couldn’t figure this out. I even attempted to write a spampd_custom rc script to include the flags I wanted, but it suffered from the same problem. Interestingly enough, if I restarted the service after the machine had booted, it would then start with the correct flags. This issue was documented in a tutorial by Technoquarter, written in 2015¹.

After asking some people in the #OpenBSD chat room on Freenode, someone dug up an old mailing list thread describing a similar problem with SpamPD and Debian² and suggested the Perl script itself may be buggy.

It took a little more searching, but I eventually found an OpenBSD mailing list thread from 2014³ which finally helped me understand what was happening. The thread indicated something I was seeing in my own logs. The SpamPD daemon was receiving a SIG HUP when running as a daemon launched at startup via the rc system.

spampd[19000]  SpamPD (type Net::Server::PreForkSimple) starting! pid(19000)
spampd[19000]: Binding to TCP port 10025 on host 127.0.0.1 with IPv4
spampd[19000]: Setting gid to "746 746"
spampd[19000]: Setting uid to "746"
spampd[19000]: Received a SIG HUP
spampd[19000]: Server closing!
spampd[19000]: Re-exec server during HUP
spampd[79539]: 2018/08/09-05:36:38 SpamPD (type Net::Server::PreForkSimple) starting! pid(79539)
spampd[79539]: Binding open file descriptors
spampd[79539]: Binding to TCP port 10025 on host 127.0.0.1 with IPv4
spampd[79539]: Setting gid to "746 746"

Apparently if SpamPD receives a Hangup Signal, it will respawn its process, but without any of the configuration flags that had been passed to it. For some reason, the OpenBSD startup scripts, when launching the SpamPD daemon, send it a SIG HUP, which doesn’t happen if you start the same service using either /etc/rc.d/spampd start or rcctl start spampd once the system is running.

The Hangup Signal is generally given to a process when it’s launched via interactive terminal (a tty) and then that terminal is lost. It’s also often used as a signal that causes a daemon to reload its configuration, so it can be updated without restarting the process.

I suspect that the OpenBSD rc system attaches pseudo-terminals to each of the daemons it runs during startup, and then disconnects them. Alternatively, it may send a SIG HUP to certain or all processes after the system is up for some reason.

I didn’t dig too deeply into the rc system to debug the issue or figure out where it may be occurring. Instead I discovered a workaround, posted in a comment⁴ in one of the tutorials I’d read previously. The simple solution is to remove spampd from the pkg_scripts variable in rc.conf.local, preventing the rc system from starting spampd at boot. Instead, simply add the following to /etc/crontab:

@reboot root /etc/rc.d/spampd start

To recap, there are two issues that lead to this problem. One is the fact that a SIG HUP causes SpamPD to restart without its command line configuration flags. The second that is that the OpenBSD startup scripts send SpamPD the Hangup Signal on boot. The successful workaround is to start spampd via the cron daemon when the system reboots.