Automated Local TLS Certificates With Step-CA

In the video below, we show you how to install and configure step-ca to automate TLS certificate provisioning for local computers

As useful as OpenSSL has been for letting me manage TLS certificates for internal IT devices, after a while this becomes time consuming and tedious

For one thing, there’s a few a steps to go through to create a certificate and just before a certificate expires you have to go through more steps to revoke the certificate and then create a new one

Now while Let’s Encrypt allows you to automate certificate provisioning, you’d need a public domain name and be willing to leak details about your internal devices to the Public Internet. In other words, it’s a bad idea

So wouldn’t it be good if you could have the security benefit of OpenSSL and the automation benefit of Let’s Encrypt

Well you can with an open source certificate authority called step-ca

Not only can you install this on an internal computer, it supports ACME provisioning

And in this video we’ll go over how to install and configure step-ca as well as demonstrate how to configure Proxmox VE to use it

Useful links:
https://smallstep.com/docs/step-ca/
https://smallstep.com/docs/step-cli/
https://hub.docker.com/r/smallstep/step-ca
https://smallstep.com/blog/private-acme-server/
https://www.cyberciti.biz/faq/linux-port-redirection-with-iptables/

Assumptions:
Because this video is specifically about step-ca, I’m going to assume that you already have a container platform like Docker or Podman installed

If not, then I do have a video which shows you how to install Podman, which is what we’ll be using in this video

Initial Setup:
The first thing to do is to initialise the setup of step-ca

Now although we’ll be doing this on Podman, the process is the same for Docker. You just need to replace the podman commands with docker ones

We’ll pull down the container image to begin with

podman pull docker.io/smallstep/step-ca

As this is a new installation and Podman is generating warnings, we’ll jump to another session where I have sudo rights and take care of that as suggested

sudo loginctl enable-linger 1002

Although you’ll probably have to change the user ID to one which is different to 1002

Now back on Podman, we’ll initilise step-ca

podman run --rm -it -v step-ca:/home/step smallstep/step-ca step ca init

This will create a volume for a step-ca container we’ll run and present us with a wizard asking a series of questions

Notice we’re using the –rm option to remove the container once it completes

This is very useful for Podman in particular which has issues if you try to run a container and the name is already used

For this example we’re choosing the following options:

Deployment type - Standalone
Name for PKI - HomeLabCA
DNS name or IP address - ca.homelab.lan
IP and port - :8443
Name of first provisioner - ca@homelab.lan
Password for CA keys and first provisioner - <your password here>

By default, the deployment mode is standalone, which is what we want

The name for this CA is entirely up to you, but I’ve opted to call it HomeLabCA

And I’m going to be connecting to this using the FQDN of ca.homelab.lan

Because this is a container, I didn’t specify an IP address, only a port. This means it will listen on port 8443 on all IP addresses, which deosn’t really matter as there is only one. In addition, it also won’t matter if the container IP address changes

I’ve opted for port 8443 as this is a non-root port and it’s slightly less obvious than the typical 443. However, you might want to pick something else to be more safe

The provisioner name is basically the admin user account, so you can name this what you like

The wizard can generate a complex password for you if you hit enter, but some special characters may cause problems later

What I tend to do is to let it create a password and replace any single or double quotes with something else, in other words, you can edit the password before you accept it

Once the process completes, copy the password and Root fingerprint and keep these somewhere safe and secure

Password File:
When the CA needs to do anything, it will require the password that was created during the setup process, but we want this to be an automated process

In which case, we’ll create a password file

To do that, we’ll run the container again but connect to it with a shell session

podman run --rm -it -v step-ca:/home/step smallstep/step-ca sh

And then we’ll create a password file using the echo command

echo -n "<your password here>" > secrets/password

I’ve used double quotes to try and deal with special characters, althouth you could use single quotes

Sometimes though you’ll find the command drops you to a > prompt. In other words, it couldn’t complete what was asked

If that’s the case then hit Ctrl-C to return to the terminal prompt

Nano isn’t installed, so one option is to use the vi text editor to create the password file instead

vi secrets/password

Press the Ins key, to go into insert mode

Now copy the password into your clipboard

Select the terminal session, right-click and select paste and you should see your password appear

Press the Ins key, to exit insert mode

Press the Esc key, press the : key, type wq in the prompt then hit return

TIP: To exit out without change, press the Esc key, then the : key, type q! in the prompt then hit return

An alternative option is to go through the initialisation again, and this time try to avoid any characters that may have caused the problem, and then allow the config files to be overwritten

In any case, check the password in the file matches the one being used

cat secrets/password

Now exit the session

exit

Bootstrapping and Testing:
The CA should now be configured, so we’ll do some initial testing

First we’ll start the container

podman run --rm -d --name step-ca -p 8443:8443 -v step-ca:/home/step smallstep/step-ca

Just make sure to use the port you chose during the initialisation, if it’s not 8443

Then check the container is running

podman ps

TIP: If you find the container isn’t staying up, chances are there’s a problem with the password in the file

You can find out more information by running the container in the foregound

podman run --rm --name step-ca -p 8443:8443 -v step-ca:/home/step smallstep/step-ca

If you’re using a software firewall on the host computer, add a rule to allow access to the CA, in this example we’ll need to allow access to TCP port 8443

In addition, you’ll want to update your DNS server to resolve the FQDN for the CA. In this example I’ll have to add a host entry for ca.homelab.lan

On another computer, we’ll install the step client software

wget https://dl.smallstep.com/cli/docs-cli-install/latest/step-cli_amd64.deb
sudo dpkg -i step-cli_amd64.deb

Then we’ll use this to bootstrap the computer so that it trusts the CA certificate

step ca bootstrap --ca-url https://ca.homelab.lan:8443 --fingerprint <your fingerprint here> --install

And now we’ll run a health check

curl https://ca.homelab.lan:8443/health

The expected output is

{"status":"ok"}

Again, make sure to use the port you choose during the initialisation, if it’s not 8443

Most, if not all, web browsers these days though have their own trust store for certificates, so you’ll need to configure them separately to trust the CA’s certificate

Now there are various ways to obtain the root certificate, but since we bootstrapped this computer, we already have easy access to it

cat .step/certs/root_ca.crt

Make a copy of that and update your web browser(s) as appropriate

Now we’ll stop the container, as there’s one more thing to do

podman stop step-ca

ACME Server Provisoner:
The main reason for wanting to use step-ca is so that we have a CA that provides automated certificate provisioning with ACME

This requires us to add an ACME provisioner to step-ca and we can do that by running the container again and issuing a command

podman run --rm -v step-ca:/home/step smallstep/step-ca step ca provisioner add acme --type ACME

The expectation is you’ll get a postive response back saying the config was updated

With the configuraton now updated, we’ll start the container back up again for actual use

podman run --rm -d --name step-ca -p 8443:8443 -v step-ca:/home/step smallstep/step-ca

At this point we’re good to go

If you plan to use the HTTP challenge option, make sure the CA can resolve the FQDNs for all of the computers that will be configured to use ACME

In addition, the CA will need access to each web server as it will perform checks

The CA will try and connect to port 80 and 443, which may require some creativity on your part if the server is listening on a different port

Proxmox VE ACME Client Configuration:
Some computer systems already support ACME whereas others will need an ACME client like Certbot installing

In this example, we’ll configure a Proxmox VE server to obtain a certificate from the CA

Now unfortunately, the GUI only allows you to register a Let’s Encrypt account so we’ll have to use the CLI to register our CA

Hopefully, Proxmox will make this possible from the GUI in a future update

Either SSH into the server or login via the GUI and open a shell session

At this stage, the server won’t trust the CA certificate so we’ll install the step client software

wget https://dl.smallstep.com/cli/docs-cli-install/latest/step-cli_amd64.deb
dpkg -i step-cli_amd64.deb

Then we’ll use this to bootstrap the computer so that it trusts the CA certificate

step ca bootstrap --ca-url https://ca.homelab.lan:8443 --fingerprint <your fingerprint here> --install

Remember to use the relevant URL for your CA

Now we’ll register an account for our CA

pvenode acme account register default testpve@homelab.lan --directory https://ca.homelab.lan:8443/acme/acme/directory

Again, set the appropriate URL and also a suitable contact name

Now, because we’re dealing with internal servers, we’ll keep things simple and use the HTTP challenge option

In other words, the CA will test HTTP(S) connectivity to our server to make sure it’s genuine before issuing a certificate

The problem is, PVE is listening on port 8006 which is not a standard HTTP/HTTPS port that the CA will try to connect on

In which case, we’ll redirect traffic from the CA using iptables

iptables -t nat -I PREROUTING --src 192.168.102.30 --dst 192.168.102.32 -p tcp --dport 443 -j REDIRECT --to-ports 8006

In this example, I want to limit access to the hypevisor, and so not only do I set the destination IP as the hypervisor, but I also set the source IP as the CA

TIP: You can check what rules like this are present with this command

iptables -t nat -L

Unfortunately rules like this don’t survive a reboot by default

In which case, we’ll we’ll edit the network interfaces file to make sure the rule persists

nano /etc/network/interfaces

And insert the following line at the end of the bridge declaration

post-up iptables -t nat -I PREROUTING --src 192.168.102.30 --dst 192.168.102.32 -p tcp --dport 443 -j REDIRECT --to-ports 8006
post-down iptables -t nat -D PREROUTING --src 192.168.102.30 --dst 192.168.102.32 -p tcp --dport 443 -j REDIRECT --to-ports 8006

Now save and exit

Now we’ll go back to the GUI and continue from there

Select the server, then navigate to System | Certificates

Under ACME, click Add

The Challenge Type can remain as HTTP

For Domain, enter the server’s FQDN, in this examples it’s testpve.homelab.lan

Now click Create

Then click Order Certificates Now

Given time, the server should be allocated a certificate and the pveproxy service will then be restarted

To trust the CA, your web browser(s) need to be updated to trust the CA

How you do that varies depending on which browser is being used

One thing to bear in mind, is that a certificate will have an expiration of 24 hours as opposed to the usual 1 year for manual certificate creation

This is deliberate because this automated solution of providing certificates doesn’t offer revocation of certificates

In theory, that’s not too bad, because if a server’s key and certificate are leaked, the certificate will likely expire before it’s had a chance to be used

And while you can’t revoke a certificate, you can block its renewal

In addition, it would probably need a man-in-the middle attack or maybe a DNS change to direct computers to a malicious server, posing as the real server

In either case, you’d have far more to worry about in those situations

Compose Service Account:
We want this container to be running all the time and for Podman, what we’ve done so far won’t survive a reboot

One option is setup a service to run the container, another is to use compose and run that as a service

Since I’ve already got a service setup to use a compose file, we’ll update that

Unfortunately though, I’ve run into problems using volumes with the version of podman-compose that is in the Debian repository so the first thing to do is to uninstall it

Using an account with sudo rights, we’ll run this command

sudo apt remove podman-compose

Next, we’ll install pipx as this is preferred by Debian for installing 3rd party Python applications

sudo apt install pipx -y

Back in a session for the podman user account we’ll update the path

pipx ensurepath

Then exit out and switch back to the account for the change to take affect

Next, we’ll install podman-compose for this user

pipx install podman-compose

Now we’ll update the compose file

nano compose.yaml

volumes:
  step-ca:
    external: true

services:
  step-ca:
    image: docker.io/smallstep/step-ca
    container_name: step-ca
    ports:
      - '8443:8443'
    restart: unless-stopped
    volumes:
      - step-ca:/home/step

Now save and exit

This is basically the same setup as the run command, but since we’ve already created the volume using the podman command, we set external to true, as podman-compose is not responsible for managing it

We’ll stop the container that’s currently running

podman stop step-ca

Then use podman-compose to spin up another instance

podman-compose up -d step-ca

As before, we’ll run a health check from our other computer

curl https://ca.homelab.lan:8443/health

Because we’ve changed the podman-compose software, I need to update the service using an account with sudo rights

sudo nano /lib/systemd/system/podman-compose.service

And change the path for podman-compose from /usr/bin/ to /home/podmanuser/.local/bin/

Bear in mind, the new path depends on the name of the user account running podman, in my case podmanuser

In other words, pipx installs software into the user’s home directory

Now save and exit

Because we’ve changed the service file details, we’ll have to reload the daemon

sudo systemctl daemon-reload

With the changes made, containers should now survive a reboot, but first we’ll check by stopping the service

sudo systemctl stop podman-compose.service

Then we’ll start it back up

sudo systemctl start podman-compose.service

We’ll check the service is still working

sudo systemctl status podman-compose.service

And check that our containers are up and running

podman ps

Summary:
Now I must admit this was a bit fiddly to setup using Podman

It should be easier to do on a default installation of Docker though, but I prefer the security of non-root containers

Do bear in mind that servers using non-standard ports add a bit more complexity to the ACME process because this is geared towards web servers using port 443

But once this is all setup, your servers should keep their certificates up to date on their own

Now, not all devices support ACME provisioning, for instance network devices, and fortunately this CA does support manual provisioning as well

So in the grand scheme of things, I expect this CA to save a lot of time going forwards

Sharing is caring!