Challenges of manual SSL certificate renewal in Kubernetes with Let's Encrypt

One of our partners runs a huge system on Kubernetes. Using Let's Encrypt, we issued SSL certificates. These certificates are valid for 90 days, then you need to renew them and issue new ones. These certificates were attached to front-end applications in Kubernetes Secrets. Due to the fact that these certificates were issued manually, they may have expired since the issuer forgot to issue a new one.

Our Kubernetes secrets are stored in gitlab. 'Sops' encrypts these secrets. This repository contained the actual certificates, and if a new certificate was issued and the issuer failed to replace the old certificate in the cluster secret but forgot to do so in the GitLab repository, the old certificate would override the new certificate when the secrets were deployed from the repository. Thereafter, the websites became unreliable and started throwing SSLHandshakeErrors. This isn't what we want.

Simplifying certificate operations: achieving automation and flexibility in Kubernetes

We wanted to find a way to issue, manage, and renew these certificates automatically to avoid the problems mentioned above. We were able to easily add new domains to handle their certificates, alongside the automaticity.

The solution

The solution’s main component is Lego. It’s a Let’s Encrypt client and ACME library written in Go.

Lego github repository

We use Lego as a Job(for creating certificates) and Cronjob(for renewing certificates) in our Google GKE cluster. We use Persistent Volumes to mount the certificates to the necessary Pods. To obtain certs we use our DNS provider. Lego has a really wide range of compatibility for DNS providers. The workflow is if we need to manage a new certificate or wildcard certificate, then first we specify a Kubernetes Job to execute Lego with the run command. Lego solves a DNS-01 challenge using the specified provider. If it has a positive outcome, then Lego creates the certificate. After that we only need to specify a Kubernetes Cronjob with the renew command to check the expiry of the certificate periodically.

The implementation

The implementation was the follows:

1. First we need to create a Persistent Volume and a Persistent Volume Claim to specify for Lego where to put certs.

2. After it, apply PV and PVC with kubectl.

3. If these are done, then we can go ahead to write our Job

apiVersion: batch/v1
kind: Job
metadata:
  name: lego-cert-creation
spec:
  ttlSecondsAfterFinished: 120
  template:
    spec:
      containers:
      - name: lego-cert-creation
        image: goacme/lego:<LEGO_VERSION>
        env:
          - name: DNSMADEEASY_API_KEY
            valueFrom:
              secretKeyRef:
                key: apiKey
                name: dnsmadeeasy-secret
          - name: DNSMADEEASY_API_SECRET
            valueFrom:
              secretKeyRef:
                key: secretKey
                name: dnsmadeeasy-secret
        args:
        - --accept-tos
        - --domains=<DOMAIN>
        - --email=<EMAIL>

ttlSecondsAfterFinished: After finishing the job, it will be deleted after 120 seconds.

• <LEGO_VERSION>: The Lego version you want to use for Lego

• Env: in the env section we use a secret we deployed to the cluster which contains the necessary data for Lego to be able to communicate with the DNS provider. We use DNSMadeEasy, but other providers may need other data to be able to communicate with their API.

• args: — accept-tos: By setting this flag to true you indicate that you accept the current Let’s Encrypt terms of service. (default: false) — domains: The domain you want to issue the certificate. If you want to issue certificates for multiple domains you just need to define a –domains argumentum for each domain. You can use a wildcard certificate here. — email: Email used for registration and recovery contact for Let’s Encrypt. — path: Directory to use for storing the data. (default: “./.lego”). It’s **/lego **since we attached the PV into that path inside the container. — dns: The DNS provider you want to use.

• run: Register an account, then create and install a certificate.

4. The Cronjob for renewing the certificate looks very similar to the Job

apiVersion: batch/v1
kind: CronJob
metadata:
  name: lego-cert-renewal-cronjob
spec:
  schedule: 1 7 * * 1
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: lego-renewal
            image: goacme/lego:<LEGO_VERSION>
            env:
              - name: DNSMADEEASY_API_KEY
                valueFrom:
                  secretKeyRef:
                    key: apiKey
                    name: dnsmadeeasy-secret
              - name: DNSMADEEASY_API_SECRET
                valueFrom:
                  secretKeyRef:
                    key: secretKey
                    name: dnsmadeeasy-secret
            args: 
            - --accept-tos
            - --domains=<DOMAIN>
            - --email=<EMAIL>

• Schedule: A cron schedule. We set the job to run every Monday at 7:01 AM.

• Args: — renew: Renew a certificate. Let’s Encrypt certificates are valid for 90 days. It renews a certificate if it expires in less than 30 days.

5. After applying Cronjob too, we must make sure that our services use these certificates. As a result, if I used the sandbox API key and secret, it did not work, so if you are implementing this solution, you should use the real credentials.

The conclusion

Using this solution, we can create and maintain SSL certificates automatically. In the future, there won’t be issues because of the expired certs. Lego is very easy to use and supports a wide range of DNS providers.

Author: Csaba Ujvári