Kubernetes requires certs on each nodes/masters to validate each other’s integrity, if the cert ever gets expired, you’d see an error like this: Unable to connect to the server: x509: certificate has expired or is not yet valid..

To fix this cluster, we first need to verify the cert status by:

$ openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver.crt
        Version: 3 (0x2)
        Serial Number: 123123123123123(0x123123123123)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
            Not Before: Nov 16 16:58:58 2017 GMT
            Not After : Nov 16 16:58:58 2018 GMT

This tells you that it expires on 2018.

Now we need to login to master nodes to reissue new cert and its related config files:

# Delete old keys
rm /etc/kubernetes/pki/{apiserver*,front-proxy-client*}
kubeadm init phase certs all --apiserver-advertise-address <ext IP> --apiserver-cert-extra-sans <int IP>
cd /etc/kubernetes/
# Delete old config
rm {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf}
kubeadm init phase kubeconfig all

After master comes back online, issue new node temp token for nodes to join: kubeadm token create.

Then on each node, delete old config and replace with kubeadm issued new configs:

mv /etc/kubernetes/manifests /etc/kubernetes/manifests.bak
rm -rf /etc/kubernetes/kubelet.conf 
rm -rf  /etc/kubernetes/bootstrap-kubelet.conf 
rm -rf /etc/kubernetes/pki/ca.crt 

kubeadm join --token=7z7kgy.bef6tsdpiyxo4xj --discovery-token-unsafe-skip-ca-verification <ext IP>:6443
mv /etc/kubernetes/manifests.bak /etc/kubernetes/manifests