Kubernetes Certificate Expiration: In-depth Analysis and Systematic Solutions

Problem Background and Symptoms

In Kubernetes cluster operations, certificate management is crucial for ensuring secure and stable operation. This article analyzes a typical failure scenario: a Kubernetes 1.6 cluster initially built on April 13, 2017, was upgraded to version 1.8 on December 13, 2017, generating new certificates, though this process may have been incomplete. By April 13, 2018, the cluster began experiencing api-server authentication errors, specifically logged as [authentication.go:64] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid].

Deep Dive into Certificate Architecture

Each node in a Kubernetes cluster contains a kubelet configuration file automatically generated by kubeadm at /etc/kubernetes/kubelet.conf. The core components of this file include the client-certificate-data and client-key-data fields, which do not point to external certificate file paths but instead embed Base64-encoded certificate and key data directly within the configuration. This design binds node authentication information closely to the configuration file, simplifying deployment but increasing complexity during certificate updates.

During initial troubleshooting, attempts were made to modify client-certificate and client-key in kubelet.conf to point to different certificate files, such as apiserver-kubelet-client.crt and apiserver.crt, but these failed because these fields should contain embedded data, not file paths. When pointing to non-existent files, system logs clearly showed invalid kubeconfig: invalid configuration: [unable to read client-cert ... due to open ...: no such file or directory] errors, confirming the unique configuration structure.

Systematic Solution Implementation Steps

Based on best practices, resolving certificate expiration requires a systematic approach rather than localized patches. The core idea is to use kubeadm to regenerate all related configuration files, ensuring the integrity and timeliness of embedded certificate data.

Certificate Inspection and Verification

First, check the expiration status of all certificates. For newer Kubernetes versions (1.15+), use kubeadm alpha certs check-expiration. For earlier versions, manually inspect with OpenSSL:

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep ' Not '

Or batch check all certificates:

find /etc/kubernetes/pki/ -type f -name "*.crt" -print|egrep -v 'ca.crt$'|xargs -L 1 -t  -i bash -c 'openssl x509  -noout -text -in {}|grep After'

Certificate Renewal and Configuration Regeneration

Certificate renewal requires step-by-step execution. First, back up existing certificates and configurations:

mv /etc/kubernetes/pki/apiserver.key /etc/kubernetes/pki/apiserver.key.old
mv /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver.crt.old
# Similarly back up other certificate files
mv /etc/kubernetes/admin.conf /etc/kubernetes/admin.conf.old
mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.old
mv /etc/kubernetes/controller-manager.conf /etc/kubernetes/controller-manager.conf.old
mv /etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.old

Then use kubeadm to regenerate certificates. Commands vary by version:

• Kubernetes 1.14 and earlier: kubeadm init phase certs all --apiserver-advertise-address <IP>
• Kubernetes 1.15-1.16: Step-by-step commands like kubeadm alpha phase certs apiserver
• Kubernetes 1.17+: kubeadm certs renew all (simplest approach)

The critical step is regenerating all kubeconfig files:

kubeadm alpha phase kubeconfig all --config /path/to/kubeadm-config.yaml

Note the use of --apiserver-advertise-address to ensure configurations contain the correct API server IP address.

Component Restart and Verification

After certificate renewal, restart related components to load new configurations:

systemctl restart kubelet
kubectl -n kube-system delete pod -l 'component=kube-apiserver'
kubectl -n kube-system delete pod -l 'component=kube-controller-manager'
kubectl -n kube-system delete pod -l 'component=kube-scheduler'
kubectl -n kube-system delete pod -l 'component=etcd'

Finally, update user kubeconfig and verify:

cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
kubeadm certs check-expiration

Version Compatibility and Best Practices

Different Kubernetes versions have significant differences in certificate management. Starting from version 1.15, kubeadm automatically updates certificates during control plane upgrades and provides kubeadm alpha certs renew for manual updates. For production environments, it is recommended to:

1. Regularly check certificate expiration times and plan updates in advance
2. Upgrade to newer versions (1.15+) to leverage automatic certificate management
3. Ensure all nodes have /etc/kubernetes/pki/ca.key for configuration generation
4. Verify kubeconfig configurations for CI/CD pipelines and other external tools after updates

The article also discusses the fundamental differences between HTML tags like <br> and character \n, where the former is an HTML structural tag for forced line breaks, and the latter is a newline character in text, requiring proper escaping in code examples based on context.