Technology

In a previous post, I introduced Bee2, a Ruby application designed to provision servers and setup DNS records. Later I expanded it using Ansible roles to setup OpenVPN, Docker and firewalls. In the latest iteration, I’ve added a rich Docker library designed to provision applications, run jobs and backup/restore data volumes. I’ve also included some basic Dockerfiles for setting up HAProxy with LetsEncrypt and Nginx for static content. Building this system has given me a lot more flexibility than what I would have had with something like Docker Compose. It’s not anywhere near as scalable as something like Kubernetes or DC/OS with Marathon, but it works well for my personal setup with just my static websites and personal projects.

Bee2 and Docker

In this iteration of Bee2, I’ve added sections in the configuration file for docker, applications, and jobs. The docker section contains general settings, such as which volumes to backup and what prefix to use for Bee2 managed containers. Both applications and jobs are for configuring Docker containers. Applications are containers that run continuously and jobs are tasks that are designed to be run once and exit, such as building content for a website.

Starting with the docker section, we have a prefix that will be appended to all the containers that are created (which defaults to bee2 if it’s omitted), and a backup section listing each server, the named volumes which should be backed up and the location to store the resulting tar files.

docker:
  prefix: bee2
  backup:
    web1:
      storage_dir: /media/backups
      volumes:
        - letsencrypt
        - logs-web

Next we have jobs and applications. Items listed in the jobs section are checked out to the local machine from a given git repository. The Dockerfile in the base of the git repository is built and run on the given machine (in this case, web1). The following example builds the static website for dyject, a Python library I wrote for dependency injection. It writes its output to a volume which is accessible by the Ngnix container as we’ll see later.

jobs:
  dyject:
    server: web1
    git: git@github.com:/sumdog/dyject_web.git
    volumes:
      - dyject-web:/dyject/build:rw

The applications section contains a list of docker applications and their configurations. Each application is given a user defined name, a build_dir (which references a directory in dockerfiles in the Bee2 source), environment variables, Docker volumes, linked containers and exposed ports.

The environment variable domains can potentially have the special keyword all, which is processed into a list of all domains being used by all applications on a given server. Because this lists needs to be passed as an environment variable to docker containers, it’s formatted as a space separated list, with each entry being a server, a colon and a coma separated lists of domains belonging to that server, as show below:

DOMAINS="bee2-app-name1:example.com,example.org bee2-app-name2:someotherdomain.com"

This highlights one of the fundamental issues with Docker, in that each container is expected to be configured using environment variables. For more complex configurations, it might make more sense to pass a JSON string as an environment variable, but that would require each container having tools within it to deserialize the passed in JSON.

The following applications section configures an HAProxy instance with publicly exposed HTTP/HTTPs ports, a Certbot to issue LetsEncrypt certificates and an Nginx instance to serve static web content:

applications:
  certbot:
    server: web1
    build_dir: CertBot
    volumes:
      - letsencrypt:/etc/letsencrypt:rw
      - /var/run/docker.sock:/var/run/docker.sock
    env:
      email: blackhole@example.com
      test: false
      domains: all
      port: 8080
  nginx-static:
    server: web1
    build_dir: NginxStatic
    env:
      domains:
        - dyject.com
      http_port: 8080
    volumes:
      - dyject-web:/www/dyject.com:ro
      - logs-web:/var/log/nginx:rw
  haproxy:
    server: web1
    build_dir: HAProxy
    env:
      domains: all
    link:
      - nginx-static
      - certbot
    volumes:
      - letsencrypt:/etc/letsencrypt:rw
    ports:
      - 80
      - 443

Bee2 communicates with Docker over the VPN tunnel that was configured in the last tutorial. Once the servers and provisioned and configured, the docker containers can be run using the following commands:

./bee2 -c conf/settings.yml -d web1:build
./bee2 -c conf/settings.yml -d web1:run
./bee2 -c conf/settings.yml -d web1:backup

To run or rebuild a specific container instead of every container listed in the configuration file, that container can be appended to the end of the command.

./bee2 -c conf/settings.yml -d web1:rebuild:haproxy
./bee2 -c conf/settings.yml -d web1:run:dyject

State information is stored using the backup command. Backups are timestamped, and running restore will pull the latest backup available in the location specified by storage_dir. An entire infrastructure stack can be rebuilt from scratch while maintaining state information by running commands like the following:

# Backup existing state
./bee2 -c conf/settings.yml -d web1:backup

# Provision and rebuild the servers
./bee2 -c conf/settings.yml -p -r

# Configure servers using Ansible
./bee2 -c conf/settings.yml -a public

# Update OpenVPN with the new keys
sudo cp conf/openvpn/* /etc/openvpn

# Restart OpenVPN (varies per Linux distribution)
sudo systemctl restart openvpn.service # systemd restart
sudo /etc/init.d/openvpn restart       # sysvinit restart
sudo sv restart openvpn                # runit restart

# Docker commands to restore state and rebuild containers
./bee2 -c conf/settings.yml -d web1:restore
./bee2 -c conf/settings.yml -d web1:build
./bee2 -c conf/settings.yml -d web1:run

A full list of Docker commands can be found by running ./bee2 -c conf/settings.yml -d help.

Under the Hood

Certbot

I’m extending the HAProxy container maintained by Docker and the official Certbot container maintained by the EFF. I try to use official containers maintained either by Docker or the project owners whenever possible. There are many HAProxy+Certbot custom container implementations currently out there, most of which place both services within the same container and then run both of them using some type of supervisor. This is necessary since HAProxy requires a signal to indicate it should reload when the SSL/TLS certificates are updated by Certbot. This seems to go against the generally accepted best-practice of isolating Docker containers to only one process.

certbot:
  server: melissa
  build_dir: CertBot
  volumes:
    - letsencrypt:/etc/letsencrypt:rw
    - /var/run/docker.sock:/var/run/docker.sock
  env:
    email: notify@battlepenguin.com
    test: false
    domains: all
    port: 8080
    haproxy_container: $haproxy

Taking a closer look at the Certbot container configured above, I place all the certificates on a volume so that they can be shared with HAProxy and be backed up. Environment variables that start with a dollar sign ($) will be replaced with the full name of a container including the prefix and container type. In this case, $haproxy will be replaced with bee2-app-haproxy. As stated earlier, the special variable all, when used with domains will be replaced with a full list of all domains associated with a given server.

In this configuration, the Docker socket is shared to the Certbot host. This is not secure nor recommended. (Although it is considered an accepted answer on StackOverflow.) If Certbot is ever compromised, an attacker could have complete access to the Docker daemon and other running containers. In this configuration, it’s essential to keep the Cerbot container up to date with the latest image to prevent any potential security vulnerabilities. In future releases, I hope to add a container with a proxy service, designed to preform necessary container tasks and provide another layer of isolation between the Docker socket and services which need to communicate to other containers.

The Docker socket is used for two things. It checks to see if HAProxy is up and running before launching Cerbot and it reloads HAProxy when certificates have been renewed. Checking to see if a container is active is done using the check_docker python script. Signaling HAProxy to reload is done by using nc like so¹:

echo -e "POST /containers/$HAPROXY_CONTAINER/kill?signal=HUP HTTP/1.0\r\n" | \
nc -U /var/run/docker.sock

HAProxy

The official HAProxy Docker image uses a systemd wrapper binary as its entry point in order to pass signals sent to the container to the underlying HAProxy process. The documentation recommends copying in a custom haproxy.cfg or using one from a mounted volume. Since I’m generating my configuration dynamically when the container starts, I had trouble figuring out the best way to run my script and still be able to forward signals to HAProxy. After a few failed attempts at trying to handle my own signals and forward them on, either in Bash or Python scripts, I eventually I settled on a custom ENTRYPOINT script that would make an exec call to the base containers ENTRYPOINT like so:

echo "Running HAProxy Config"
python /haproxy-config.py

echo "Running Base Wrapper Script"
exec /docker-entrypoint.sh "$@"

This allows me to run my custom haproxy-config.py script which sets up virtual hosts, SNI and Certbot, and then exec out to the docker-entrypoint.sh. This replaces my process with the stock binary signal manager that comes with the official Docker container. I call my startup script within the Dockerfile like so:

ENTRYPOINT ["/startup"]
CMD ["haproxy", "-f", "/usr/local/etc/haproxy/haproxy.cfg"]

Nginx

All my static content is hosted via nginx. The HAProxy script will create vhost/SNI entries for every domain in the domain map passed to it, using port 8080 on the destination container. HAProxy handled SSL redirects and offloads the SSL traffic. The following configuration, modified from a stackoverflow answer, will atomically direct virtual host requests to an appropriate folders located at /www/<domain name> and issue 301 redirects from the www subdomain back to the root. The option port_in_redirect off needs to be used to ensure nginx doesn’t add port 8080 to folder name redirects.

server {
  listen 8080;
  server_name ~^(www\.)(?<domain>.*)$;
  return 301 https://$domain$request_uri;
}

server {
    listen 8080 default_server;

    server_name _default_;
    root "/www/$http_host";
    port_in_redirect off;

    access_log "/var/log/nginx/$http_host.log" main;

    location ~ /\.ht {
        deny all;
    }
}

The nginx Dockerfile is fairly straightforward as well. By default, the base nginx container sends all of its access and error logs to /dev/stdout and /dev/stderr as it is considered best practice with Docker. I’ve modified nginx’s configuration to keep separate log files for each individual host, so that they can be run through a log parser later for analytic purposes. Since nginx worker threads run as an unprivileged user within the container, by default nginx doesn’t have permission to write to its own log directory. However, if permissions are set in the Dockerfile for the directory that the Docker log volume will be mounted into, those permissions carry over to the mount itself, allowing nginx to store its logs and also have those logs available to other containers.

FROM nginx:1.13.6

COPY nginx.conf /etc/nginx/nginx.conf
RUN chown -R nginx:nginx /var/log/nginx

EXPOSE 8080

VOLUME ["/var/log/nginx", "/var/www"]

IPv6 Support

By default, Docker’s exposed ports do not listen on both the IPv4 and IPv6 addresses of the host. Configuring Docker to listen on IPv6 involves giving the Docker daemon a slice of the host’s subnet². I acomplished this by taking the subnet provided to me by my hosting provider, adding a /80 to it and adding it to the daemon.json configuration file using the existing Ansible role and template:

{
	"tls": true,
	"tlsverify": true,
	"tlscacert": "{{ docker_ca }}",
	"tlscert": "{{ server_crt }}",
	"tlskey": "{{ server_key }}",
	"ipv6": true,
	"fixed-cidr-v6": "{{ state.servers[ansible_hostname].ipv6.subnet }}/80",
	"hosts": ["127.0.0.1:2376", "{{ private_ip }}:2376", "fd://"]
}

Final Thoughts

Bee2 was never really meant to a general purpose provisioning system. I attempted to use existing tools such as Terraform and Docker Compose, but had trouble with some of their limitations. It’s primary purpose was to aid in the migration of my personal websites and projects from my existing hosting provider, which requires considerable manual configuration, to an automated provisioning system. Although only targeting one provider currently, creating my services in code should ease in further migrations, as well as quickly spinning up new applications to experiment with.

The initial Docker work was based in part of the frame work used in my side project BigSenseTester, where I use a Ruby script to create Docker containers and run the automated integration tests for BigSense. The vSense project also had configuration management tools for HAProxy and Certbot. Although I was able to leverage some of this existing work, there was still a considerable amount I needed to add and adapt for this particular iteration of Bee2.

Working on Bee2 has given me a deep appreciation to all the intricacies involved in writing provisioning services and automating configuration management. Although I’ve gained a lot of flexibility by writing a custom application to take care of my specific desires in a provisioning tool, it comes at the expense of the time needed to develop it, instead of the future projects I hope to host with it. Hopefully the work I’ve done can help others in developing their own development and operation tools, as well as speed up development of my own projects in the future.