Administration

How to use the systems that already exist, and how to rotate or replace things on them as needed.

Deployment tooling

Currently deployment is a very manual process. This repo provides two scripts rebuild.sh and install.sh which can be called using ./rebuild.sh and ./install.sh respectively.

The rebuild script is intended to be used to update a currently running system. To see its full set of options you can run ./rebuild.sh -h

See the contributing documentation for how the install script is used to set up a new system.

In the future we will change our deployment strategy so that we have a build server that automatically builds new images on update to main and then push those updates to available devices and make images available for devices to pull when they become available.

Updating dependencies

You can update configuration dependencies using npins update.

Rotating secrets

Certificates

Every proxy in the fleet gets its tls certificates from Let’s Encrypt on its own. Beacon asks for one certificate per public name, proven over http. Defiant asks for one wildcard certificate for jan-leila.com, proven by writing a dns record into the challenge zone that beacon serves.

All certificate automatically renew themselves.

Beacon serving the challenge zone means it could answer the wildcard’s challenge itself. To prevent this a CAA record binds wildcard issuance to defiant’s acme account and plain issuance to beacon’s. The certificate authority then refuses anything else. These records live where the public zone is hosted and are set by hand.

The account urls are generated on each node, once it has been issued anything:

sudo find /var/lib/acme/.lego -name account.json -exec grep -ho 'https://[^"]*/acme/acct/[0-9]*' {} +

From there set these DNS records.

jan-leila.com. CAA 0 issuewild "letsencrypt.org; accounturi=<defiant>"
jan-leila.com. CAA 0 issue     "letsencrypt.org; accounturi=<beacon>"

A node’s account changes if its acme state directory is ever lost.

Metrics

Currently metrics are not yet implemented. More to come on this in the future. See the metrics module docs for more details on the plans.