We have once again been having issues with LinkerD. Seems this happens every couple years. This time we saw that our pods were failing to start. This included the linkerd pods themselves. We saw this in both our dev and main clusters so it must be something with our setup. At first restarting the identity and/or destination pods (not sure which did it) would fix it but yesterday not even that worked on our dev cluster. I had to remove everything including the certs and have ArgoCD reinstall everything. Now I am trying to figure out if my main cluster is going to break in a couple days if I don’t recreate the certs.
The error I am seeing is this:
level=fatal
msg=“Failed to initialize identity service: failed to verify issuer credentials for ‘identity.linkerd.cluster.local’ with trust anchors: x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "root.linkerd.cluster.local")”
The weird part is that none of our certs were anywhere close to being expired. I have them set to expire in a VERY long time. Cert-Manager and Trust-Manager should be managing everything and renewing everything.
I am not sure where to even look to see if our Main cluster will have a problem. For now we are just meshing a single namespace and waiting some time to see if issues crop up but I would really like to build trust so that we can remesh our prod pods.
Any insights would be helpful.