How to rotate bosh-dns certificates in TKGI 1.9+ using maestro

This procedure can be used to rotate the bosh-dns certificates which are all signed by the CA /opsmgr/bosh_dns/tls_ca. These certificates are non-configurable leaf certificates. Later in the post we will walk through how the rotation of these certificates can be verified. Followed by a walk through of garbage collecting the old certificates. More details on the certificate types can be found here.

Gather certificate information

Setup credentials for credhub and maestro access

export BOSH_CLIENT=ops_manager BOSH_CLIENT_SECRET=<secret>
export BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate BOSH_ENVIRONMENT=<IP>

export CREDHUB_SERVER="$BOSH_ENVIRONMENT:8844" CREDHUB_CLIENT="$BOSH_CLIENT"
export CREDHUB_SECRET="$BOSH_CLIENT_SECRET" CREDHUB_CA_CERT="$BOSH_CA_CERT"

credhub api https://$BOSH_ENVIRONMENT:8844 --ca-cert=/var/tempest/workspaces/default/root_ca_certificate
credhub login

You can use the below command to get a list of certificates signed by /opsmgr/bosh_dns/tls_ca

maestro --json topology --name /opsmgr/bosh_dns/tls_ca | jq .topology[].signs[].name

"/bosh_dns_health_client_tls"
"/bosh_dns_health_server_tls"
"/dns_api_client_tls"
"/dns_api_server_tls"

Check the current validity of the certificates

maestro --json topology --name /opsmgr/bosh_dns/tls_ca | jq '.topology[].signs[] | "\(.name) \(.versions[].valid_until)"'

"/bosh_dns_health_client_tls  2021-12-18T02:30:06Z"
"/bosh_dns_health_server_tls  2021-12-18T02:30:06Z"
"/dns_api_client_tls  2021-12-18T02:30:07Z"
"/dns_api_server_tls  2021-12-18T02:30:07Z"

How to check which deployments are using these certificates

The bosh-dns certificates are used by almost every deployment. In the example below, there are only two deployments as this environment only has TKGI installed along with one K8s cluster.

maestro --json topology --name /opsmgr/bosh_dns/tls_ca | jq '.topology[].signs[] | "\(.name)  \(.versions[].deployment_names)"'

"/bosh_dns_health_client_tls  [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"]"
"/bosh_dns_health_server_tls  [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"]"
"/dns_api_client_tls  [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"]"
"/dns_api_server_tls  [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"]"

Certificate Rotation Procedure

The certificates can be rotated using the maestro utility. Since all these certificates are signed by the same CA, we will leverage this fact to regenerate them all at once. The certificates that are to be regenerated can be confirmed via a dry-run first. In the output below we see only four certificates being regenerated

maestro regenerate leaf --signed-by /opsmgr/bosh_dns/tls_ca --dry-run

to_be_regenerated:
    - name: /bosh_dns_health_client_tls
      certificate_id: 0b402fa6-04a3-492f-849d-cde95f5cff88
    - name: /bosh_dns_health_server_tls
      certificate_id: b6e7b8f2-edb4-4c2e-a1a5-332cd9f28c37
    - name: /dns_api_client_tls
      certificate_id: 0ae32163-a76a-4d14-8fa8-79e5402b9511
    - name: /dns_api_server_tls
      certificate_id: 5ac68fdb-01ef-4d50-a9f3-92281e57a74c

Regenerating Certificates

Following command can be used to regenerate the certificate. Upon successful regeneration you'll see a similar output

maestro regenerate leaf --signed-by /opsmgr/bosh_dns/tls_ca

regenerated:
    - name: /bosh_dns_health_client_tls
      certificate_id: 0b402fa6-04a3-492f-849d-cde95f5cff88
    - name: /bosh_dns_health_server_tls
      certificate_id: b6e7b8f2-edb4-4c2e-a1a5-332cd9f28c37
    - name: /dns_api_client_tls
      certificate_id: 0ae32163-a76a-4d14-8fa8-79e5402b9511
    - name: /dns_api_server_tls
      certificate_id: 5ac68fdb-01ef-4d50-a9f3-92281e57a74c

After regeneration, querying these certificates using maestro, shows two copies(old and new) of each certificate. This also confirms that a new certificate was added to credhub. The old and new certificates can be identified using the deployment_names and timestamp as shown in the output below

[OLD]
maestro --json topology --name /opsmgr/bosh_dns/tls_ca | jq '.topology[].signs[] | "\(.name) \(.versions[1].deployment_names) \(.versions[1].valid_until)"'

"/bosh_dns_health_client_tls [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"] 2021-12-18T02:30:06Z"
"/bosh_dns_health_server_tls [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"] 2021-12-18T02:30:06Z"
"/dns_api_client_tls [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"] 2021-12-18T02:30:07Z"
"/dns_api_server_tls [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"] 2021-12-18T02:30:07Z"

[NEW]
maestro --json topology --name /opsmgr/bosh_dns/tls_ca | jq '.topology[].signs[] | "\(.name) \(.versions[0].deployment_names) \(.versions[0].valid_until)"'

"/bosh_dns_health_client_tls [] 2021-12-18T06:18:35Z"
"/bosh_dns_health_server_tls [] 2021-12-18T06:18:35Z"
"/dns_api_client_tls [] 2021-12-18T06:18:35Z"
"/dns_api_server_tls [] 2021-12-18T06:18:35Z"

Although these certificates have been regenerated and added to credhub, the new certificates are not deployed to the actual VMs yet. This is evident from the blank([ ]) deployment name assigned to the new certificates.

Deploying the certificates

After the certificates have been regenerated, Apply Changes is needed to push these certificates to the VMs. From the Opsmgr UI

• Select Review Pending Changes
• Select TKGI tile
• Select upgrade all clusters errand
• If you can't rotate these certificates on all the clusters and prefer to deploy one cluster at a time pks upgrade-cluster <cluster-name> can be used via cli to achieve this.
• Wait for above operation(s) to complete

How to verify certificate rotation

After finishing the previous step, running the command below, we can see that the new certificates are associated with the deployments now

[OLD]
maestro --json topology --name /opsmgr/bosh_dns/tls_ca | jq '.topology[].signs[] | "\(.name) \(.versions[1].deployment_names) \(.versions[1].valid_until)"'

"/bosh_dns_health_client_tls [] 2021-12-18T02:30:06Z"
"/bosh_dns_health_server_tls [] 2021-12-18T02:30:06Z"
"/dns_api_client_tls [] 2021-12-18T02:30:07Z"
"/dns_api_server_tls [] 2021-12-18T02:30:07Z"

[NEW]
maestro --json topology --name /opsmgr/bosh_dns/tls_ca | jq '.topology[].signs[] | "\(.name) \(.versions[0].deployment_names) \(.versions[0].valid_until)"'

"/bosh_dns_health_client_tls [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"] 2021-12-18T06:18:35Z"
"/bosh_dns_health_server_tls [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"] 2021-12-18T06:18:35Z"
"/dns_api_client_tls [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"] 2021-12-18T06:18:35Z"
"/dns_api_server_tls [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"] 2021-12-18T06:18:35Z"

Cleaning up inactive certificates

After successful regeneration, the older inactive certificates can now be garbage collected. This can be done using the commands below.

Note: The gc command also has a --name option which can be used to garbage collect a single certificate

maestro gc leaf --all
deleted:
    - name: /dns_api_server_tls
      certificate_id: 5ac68fdb-01ef-4d50-a9f3-92281e57a74c
      version_ids:
        - 377142ec-578d-4df7-8abe-1db375cd380a
    - name: /bosh_dns_health_server_tls
      certificate_id: b6e7b8f2-edb4-4c2e-a1a5-332cd9f28c37
      version_ids:
        - 529bad95-1fd0-4469-9fb5-990b40453a13
    - name: /bosh_dns_health_client_tls
      certificate_id: 0b402fa6-04a3-492f-849d-cde95f5cff88
      version_ids:
        - 0c93c743-0dd2-477a-8771-9773d8065198
    - name: /dns_api_client_tls
      certificate_id: 0ae32163-a76a-4d14-8fa8-79e5402b9511
      version_ids:
        - 8c995186-2f79-4c0a-ba40-02dc2805b32f

Confirm that the inactive certificates are deleted.

maestro --json topology --name /opsmgr/bosh_dns/tls_ca | jq '.topology[].signs[] | "\(.name) \(.versions[0].deployment_names) \(.versions[0].valid_until)"'

"/bosh_dns_health_client_tls [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"] 2021-12-18T06:18:35Z"
"/bosh_dns_health_server_tls [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"] 2021-12-18T06:18:35Z"
"/dns_api_client_tls [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"] 2021-12-18T06:18:35Z"
"/dns_api_server_tls [\"pivotal-container-service-3b9cfff74271c08d9e0d\",\"service-instance_16d9b7c4-c1fe-4be9-81c3-0aedda0ea6c0\"] 2021-12-18T06:18:35Z"