Automate Certificate Management With Caddy

In the video below, we show you how to automate certificate management using Caddy and ACME with a DNS challenge

Now I don’t know about you but when it comes to IT, I don’t look forward to manually managing TLS certificates

But certificate management is important for security

Fortunately for public facing servers at least, Let’s Encrypt came along and they gave us automated certificate management

And while it could be used for internal servers as well, it’s a terrible idea to provide your private IT information to a 3rd party public server

Fortunately, Caddy can be configured as an ACME server on your private network

Granted you still have to configure your computers to trust it’s root certificate, but once that’s done, your internal computers will have automated certificate management

But how do you configure Caddy to act as an ACME server, and how do you integrate this with a DNS server like Bind9 to support DNS challenges?

Useful links:
https://caddyserver.com/docs/caddyfile/directives/acme_server
https://caddyserver.com/docs/caddyfile/options
https://caddy.community/t/use-caddy-for-local-https-tls-between-front-end-reverse-proxy-and-lan-hosts/11650
https://bind9.readthedocs.io/en/latest/reference.html#dynamic-update-policies
https://smallstep.com/docs/tutorials/acme-protocol-acme-clients/#popular-acme-clients

Assumptions:
Now because this video is about configuring Caddy as an ACME server, I’m going to assume you already have Caddy installed or you at least know how to do that

If not, then I do have another video which shows you how to install Caddy as a container in Docker

I’m also going to assume you have a DNS server like Bind9 that supports Dynamic Updates or you at least know how to set this up

If not, then I do have another video that shows you how to do this on a Linux server

And I have another video which shows you how to do this in a container

Although if you have another DNS server that supports Dynamic DNS, feel free to use that instead

Overview:
Now before we get started with any configuration work I think it makes sense to go over the basics of ACME and explain why we’ll be using a DNS challenge

So first of all, what is ACME?

Well, ACME stands for Automatic Certificate Management Environment which is what Let’s Encrypt came up with to make provisioning TLS certificates sooo much easier

Basically, a web server for instance, requires a TLS certificate, and it uses an ACME client to ask the ACME server to provide it with one

The ACME server needs to validate that request because you can’t have everybody spinning up web servers claiming to belong to microsoft.com for instance

There are two direct methods to do that, but usually in an internal network you’ll have devices behind firewalls and they may not even be listening on the standard ports of 80 and 443 used by HTTP and HTTPS

So in this video, we’ll use a DNS challenge instead

For this, the ACME server will tell the web server it needs to see a text record on the domain’s DNS server which contains a specific value

To fully automate this, the DNS server will have to support Dynamic DNS and the web server in this case will need to be able make DNS changes

Assuming the ACME server sees the text record in DNS, it will then have its validation and it can then issue a certificate to the web server

It’s then left to the web server to load the certificate and it will then have to regularly repeat this process before the certificate expires

Dynamic Updates:
Because we’re using a DNS challenge, Bind9 needs updating to support this for ACME clients

We don’t want just any device being able to make changes so we’ll create a key or passphrase to restrict this ability

TIP: The developers of Bind9 removed IP address restriction because the source IP address can be spoofed

Now depending on your environment you could give all your devices the same key and allow them to make changes for any server, or you could lock this down so each device has its own key and can only update its own records

A “global” key if you will is much simpler to set up but it will require more work later on if a server gets compromised, because then every server will need to be updated

A “specific” key on the other hand requires a lot more work now, but it’s more secure and less work will be needed if a server is later compromised

For this example, we’ll create one key for all ACME clients to use, but different to the one used by DHCP servers

For Debian, the command we need is installed as part of the Bind installation

tsig-keygen -a hmac-sha256 acme.key | tee -a bind9/etc/bind/dns.keys.conf acme.key

I’m running Bind as a container though so I’ll need to run this within the container

docker exec bind9 tsig-keygen -a hmac-sha256 acme.key | tee -a bind9/etc/bind/dns.keys.conf acme.key

cat acme.key

key "acme.key" {
	algorithm hmac-sha256;
	secret "i7DjNuAZYanLNCYOW3HVIEQRXlNJx3dy3d8AU8Cenp0=";
};

But I want to store this in a file for Bind to read

For me, a file already exists so I’ve piped the output to that file using tee with the -a option to append

I also want to do some testing, so the output will also be sent to another file I’ve called acme.key

The reason I’ve done this is because we’ll later use the nsupdate command and as far as I’m aware, I can’t tell it which key to choose within a file

TIP: Although the documentation mentions using an FQDN for the name, it doesn’t seem to be necessary, although it would make it easier to identify what a key is for

We need to update Bind9 with an extra policy command for ACME clients

nano bind9/etc/bind/named.conf.local

Now it looks a bit like this

include "/etc/bind/dns.keys.conf";

zone "homelab.lan"  {
	type master;
	file "/var/lib/bind/db.homelab.lan";
        update-policy { 
		deny * name pvedemo1.homelab.lan;
		grant dhcp.key wildcard *.homelab.lan A DHCID;
        grant acme.key zonesub TXT;
	};
};

...

Now save and exit

I already have the file dns.keys.conf referenced in this file and it’s also mapped to the container

In which case, adding an extra key doesn’t require an update to the compose file either

I already have a policy to prevent updates for one specific record, another to let DHCP servers make changes and now I’m adding an extra line at the end for ACME clients

With this new policy, devices with the key can add and remove text records in the homelab.lan zone, plus sub zones

And we allow sub zones because of the way in DNS challenges work

For example, if a server called www.homelab.lan asks for a certificate, the ACME server will ask to see a text record called _acme-challenge.www.homelab.lan

To DNS that means a record called _acme-challenge in the www subdomain or sub zone of homelab.lan.

We’ll stop and start the container again to make sure these changes take effect

docker compose down bind9
docker compose up -d

Then check the container is running

docker ps -l

Before move on we should check DNS still works as a simple typo could break things

host testdns1.homelab.lan
host www.microsoft.com

NSUPDATE:
Now before we move on to configuring ACME, we should first test to see if Dynamic Updates work

Otherwise, if we have problems with Caddy, it’s harder to know where the problem is

And we can do this using the nsupdate command

I’ll switch to a user with privileges first

Then we’ll make sure the dnsutils package is installed

sudo apt update
sudo apt install dnsutils -y

Now I’ll switch back to our Docker user

Next, we’ll create an instruction file

nano acmetest.txt

server testdns1.homelab.lan
update delete _acme-challenge.test1.homelab.lan. TXT
update add _acme-challenge.test1.homelab.lan. 600 TXT "Test record"
show
send

Now save and exit

First we want to pick the correct DNS server

Then we’ll delete an existing TXT record, if it exists, and then add a new TXT record

We’ll then display the changes and apply them

Now we can test this

nsupdate -k acme.key -v acmetest.txt

The -v option incidentally is to make sure TCP is used, which is better than using UDP for large changes. But it’s easier to just to stick to using TCP

Assuming there are no errors, we’ll check we can find this record

dig -t TXT _acme-challenge.test1.homelab.lan.

This should return a status of NOERROR and we should see the new record

Now, my concern here is anything with this key can create any TXT record within the zone

If you’re really security conscious, you could lock this down so a server can only change it’s own ACME related TXT record

That would require a lot of extra work, and it’s beyond the scope of this video as it relates more to DNS security

Anyway, now that we know Bind9 is ready, we’ll tidy things up

nano acmetest.txt

server testdns1.homelab.lan
update delete _acme-challenge.test1.homelab.lan. TXT
show
send

Now save and exit

Then run the command to remove the TXT record

nsupdate -k acme.key -v acmetest.txt

We should also remove the temporary files

rm acmetest.txt
rm acme.key

New Root Certificate:
Now this next step isn’t necessary, but a problem I find is that each instance of Caddy has the same name on the root certificate

And if you’re running separate instances of Caddy on your network it becomes difficult to know which one is which in your browser

So now is as good a time as necessary to change that name

WARNING This will remove all existing certificates and keys that are stored in the Caddy container

If you do this, you’ll have to update every device to trust a new root CA

As long as they trust the old root certificate, any existing server certificate should still work until it expires

But new certificates that are created will be signed by a new root CA

And bear in mind that the root certificate will expire in 10 years anyway, so it helps to be prepared…just in case

The first thing we’ll do is to edit the Caddyfile

nano caddy/Caddyfile

Then we’ll paste this at the top

{
	pki {
		ca local {
			name "Lab Demo CA"
		}
	}
}

This is how we define global settings for Caddy, in this case we’re changing some PKI settings

NOTE: You need to use tabs and not spaces in the formatting

While you can configure multiple CAs on one server, and in an example the developers do that, I want to keep things simple so we’re renaming the local CA

The simplest way I can think of is to force a rebuild of the existing caddy-data volume

docker compose down caddy
docker volume ls
docker volume rm dockermgr_caddy_data
docker compose up -d

Each volume usually contains the Docker user’s name but I’ve opted to list all volumes so I can check what the actual name is before deleting anything

When Caddy starts back up, Docker will create a new data volume for it and Caddy will have to create new certificates and keys for the root and intermediate CAs as well as for servers

But we’ll check Caddy is actually still working

docker ps -l

We now need a copy of the new root certificate so we’ll copy that to the host

docker compose cp caddy:/data/caddy/pki/authorities/local/root.crt caddy/lab-demo-root.crt

You should then import this into your browser for instance, and/or update OS trust stores with it

How you do that depends on the web browser and operating system

I tend to find that web browsers typically need a refresh for the changes to take effect; Ctrl-F5

On some occasions though I’ve had to restart the web browser

Caddy ACME Server:
Setting up an ACME server in Caddy is actually quite simple

We need to add a new web server entry to the Caddy file, but one that will act as an ACME server

nano caddy/Caddyfile

acme.homelab.lan {
	tls internal
	acme_server {
		challenges dns-01
		allow {
			domains *.homelab.lan
			ip_ranges 192.168.102.0/24
		}
	}
}

Now save and exit

I’m giving it a hostname of acme, but you can call it something else and my lab domain is called homelab.lan

We want this session to be secure, so we tell Caddy to provide a TLS session and to serve the certificate from its own internal CA

By default, Caddy will try to use an HTTP or TLS-APN challenge

These are both direct access methods, so in the example I’ve set the challenge type to DNS only

I’m restricting the ACME server to only provide certificates for this lab domain and to only allow access for clients in a specific IP range

TIP: If you want to add more subnets to the line, you can separate them out with a space

Granted, our DNS server will only allow updates to this same domain anyway, but when it comes to security, more layers can be better

Feel free to restart the container

docker compose restart caddy

Otherwise you can instruct caddy to reload its config file

docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile

Either way, it’s best to check Caddy is still working

docker ps -l

TIP: If you ever need shell access to the Caddy container, run this command

docker compose exec --user=root caddy sh

This can be useful if you want to troubleshoot issues within Caddy itself

Because we’ve created a new web server here, you should also update your DNS server as clients need to resolve this new FQDN

In my case, I’ll need to add an A record for acme.homelab.lan to Bind9

nano nsupdate.txt

server testdns1.homelab.lan
update delete acme.homelab.lan. A
update add acme.homelab.lan. 600 A 192.168.102.30
show
send

Now save and exit

TIP: I don’t add a PTR record as this is a container and the IP address already relates to the Docker host

nsupdate -k ddns.key -v nsupdate.txt

My ACME clients will then need to use this URL to request a certificate

https://acme.homelab.lan/acme/local/directory

And I can check this at least responds with a computer that trusts Caddy’s new root certificate

curl https://acme.homelab.lan/acme/local/directory

As long as this doesn’t return an error, you should be able configure ACME clients

Summary:
For me, Caddy now acts as a reverse proxy for my containers and it automates certificate management for computers that support ACME

Even some other Caddy instances can get their certificate from this Caddy server

In theory, I could extend this further and have Caddy provide manually applied certificates

In other words, there are some devices that can only be updated manually

Practically, these certificates would need a longer lifetime of maybe 1 year

And although Caddy did have a bug when I last checked, hopefully that’s now fixed and we’re no longer limited to 12 hour certificates

That would then leave me needing only one root certificate to access ALL of my internal devices

Sharing is caring!