Rotating Consul TLS Certificates¶

Consul uses mutual TLS for traffic between its agents and for the VOR Stream services that query it, such as Vault, RabbitMQ, Super, the SDK service and Django. These certificates are separate from the ones that tls_ca_cert_file_path, tls_cert_file_path and tls_key_file_path name in the inventory file: replacing the files in /opt/vor/tls does not replace them. Once a Consul certificate expires, Consul rejects the connections that present it, and the services that depend on Consul fail with errors that mention an expired certificate.

The paths on this page assume the default vor_root of /opt/vor, datacenter dc1 and domain consul. Run the commands as the installer account, e.g., vrisk, in a login shell (sudo -iu vrisk), so that the profile script in /etc/profile.d puts consul, vault and supervisorctl on the PATH.

Certificates and Their Lifetimes¶

Every file lives in /opt/vor/etc/consul.d.

Files	Hosts	Lifetime
`consul-agent-ca.pem`	Every host	Fixed when the CA is created; `consul tls ca create` defaults to five years
`dc1-server-consul-0.pem`, `dc1-server-consul-0-key.pem`	Hosts in the `consul` group	One year
`dc1-client-consul-0.pem`, `dc1-client-consul-0-key.pem`	Every host	One year

consul-agent-ca.pem is the CA that consul_ca_cert names in the inventory. Every deployment run without --limit deletes the server and client certificates on each host and issues new ones signed by that CA, using the key that consul_ca_key names. The CA key is copied to a host only while its certificates are issued, then removed. An installation deployed at least once a year therefore never sees its server and client certificates expire. The CA is never reissued: it expires on the date it was created with, however often the installation is deployed.

The packaged CA

The hosts.ini in the deployment package points consul_ca_cert and consul_ca_key at a CA bundled with the package. Every installation that keeps the default shares that CA, and anyone with a copy of the package holds its key. It expires on May 3, 2027, and every installation still using it stops working then. Create a CA for each installation, as described in Replacing the CA.

Checking When the Certificates Expire¶

On each host, print the expiry date of every Consul certificate:

$ cd /opt/vor/etc/consul.d
$ for cert in consul-agent-ca.pem dc1-*-consul-0.pem; do echo "$cert: $(openssl x509 -noout -enddate -in "$cert")"; done
consul-agent-ca.pem: notAfter=May  3 01:11:17 2027 GMT
dc1-client-consul-0.pem: notAfter=Oct  1 06:59:04 2027 GMT
dc1-server-consul-0.pem: notAfter=Oct  1 06:59:04 2027 GMT

To alert ahead of expiry, openssl x509 -checkend exits with a non-zero status when a certificate expires within the given number of seconds. This checks for the next 30 days:

openssl x509 -noout -checkend 2592000 -in /opt/vor/etc/consul.d/dc1-client-consul-0.pem

Rotating the Server and Client Certificates¶

Check the CA first. If consul-agent-ca.pem has expired, new certificates signed by it are no better than the old ones, and a deployment copies it back onto every host; replace the CA instead.

Rotating by Redeploying¶

Rerun the deployment with the same inventory, as described in Playbook Execution. It issues new server and client certificates on every host and restarts Consul. Then restart the services on every host.

Note

A deployment restarts Consul, but not necessarily every service that connects to it. Those services read the client certificate when they start, so one left running keeps presenting the old certificate, and fails once it expires.

Rotating by Hand¶

When the deployment cannot be run, repeat on each host what it does. Start with the CA certificate and key that consul_ca_cert and consul_ca_key name in the inventory. A relative path there is relative to the consul directory of the deployment package, so the defaults are consul/artifacts/consul-agent-ca.pem and consul/artifacts/consul-agent-ca-key.pem. The examples use an inventory of two hosts: consul1.example.com in the consul group and compute1.example.com.

On every host, stop all services:
```
supervisorctl stop all
```
On every host, copy the CA certificate and key into the Consul configuration directory as consul-agent-ca.pem and consul-agent-ca-key.pem. The profile script points every user's vor commands at the CA certificate, so it must be readable by all. Run the remaining steps from that directory.
```
cd /opt/vor/etc/consul.d
cp /path/to/consul-agent-ca.pem consul-agent-ca.pem
cp /path/to/consul-agent-ca-key.pem consul-agent-ca-key.pem
chmod 644 consul-agent-ca.pem
chmod 600 consul-agent-ca-key.pem
```
On each host in the consul group, delete the server certificate and issue a new one. Pass -additional-dnsname once for every host in the consul group, as a deployment does:
```
rm -f dc1-server-consul-0.pem dc1-server-consul-0-key.pem
consul tls cert create -server -dc=dc1 -domain=consul -additional-dnsname=consul1.example.com
```
On every host, delete the client certificate and issue a new one. Pass -additional-dnsname once for every host in the inventory:
```
rm -f dc1-client-consul-0.pem dc1-client-consul-0-key.pem
consul tls cert create -client -additional-dnsname=consul1.example.com -additional-dnsname=compute1.example.com
```
Important

Delete the old files first. consul tls cert create never overwrites a certificate: when dc1-client-consul-0.pem exists, it writes dc1-client-consul-1.pem instead, which nothing reads.
On every host, delete the CA key and make the client key readable by every user, for the same reason as the CA certificate.
```
rm consul-agent-ca-key.pem
chmod 644 dc1-client-consul-0-key.pem
```
Start all services, on the hosts in the consul group first, then on the rest:
```
supervisorctl start all
```
Unseal Vault.

Restarting Services¶

On every host, starting with the hosts in the consul group, restart all services so that each one reads the new client certificate:

supervisorctl restart all

Then unseal Vault.

Unsealing Vault¶

Restarting Vault seals it, and services cannot read their secrets until it is unsealed. Skip this step if Vault is managed outside the deployment, i.e., the inventory has no vault hosts.

The deployment stores five unseal keys in vault_storage_path, which defaults to /opt/vor/vault/data, on the host that initialized Vault, normally the first host in the vault group. Any three of them unseal Vault. On that host, point VAULT_ADDR at each Vault host in turn, using that host's vault_listener_address (its inventory hostname on port 8200 by default, with http:// if Vault is deployed without TLS), and unseal it. If the profile script does not already set VAULT_CACERT, set it to the file that tls_ca_cert_file_path names in the inventory, so that vault can verify a certificate signed by a private CA:

export VAULT_ADDR=https://vault1.example.com:8200
export VAULT_CACERT=/opt/vor/tls/ca.pem
vault operator unseal "$(cat /opt/vor/vault/data/unsealKey0)"
vault operator unseal "$(cat /opt/vor/vault/data/unsealKey1)"
vault operator unseal "$(cat /opt/vor/vault/data/unsealKey2)"

A service that reads secrets from Vault can use up its start retries while Vault is sealed. Once every Vault host is unsealed, check supervisorctl status on each host and start any service that is not RUNNING. See Management of Services.

Replacing the CA¶

Replace the CA before it expires, or to stop using the packaged CA. Every host must switch at once: a host on the old CA cannot talk to a host on the new one, so Consul and the services that depend on it are unavailable until every host has been restarted.

Create a CA in an empty directory on any host with the consul binary, such as a deployed host. This writes consul-agent-ca.pem and consul-agent-ca-key.pem to the current directory, and refuses to if either file already exists there; -days sets the CA's lifetime:
```
mkdir ~/consul-ca && cd ~/consul-ca
consul tls ca create -days=1825
```
Keep the key secret. Anyone who holds it can issue certificates that Consul trusts.
Copy both files to the machine you run Ansible from, which is where the deployment reads them, and point consul_ca_cert and consul_ca_key in the inventory at them.
Rerun the deployment, then restart the services on every host.

To replace the CA without a deployment, follow Rotating by Hand with the new CA's files, and still point the inventory at them: the next deployment otherwise copies the old CA back onto every host.