Hi @Flynn , We faced another instance of this same problem in another cluster this time there was no recent cert renewal for any controller pods.
[706995.441763s] ERROR ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_proxy_identity_client::certify: Failed to obtain identity error=status: Unknown, message: "controller linkerd-identity-headless.linkerd.svc.cluster.local:8080: endpoint 10.200.39.163:8080: connection error: received fatal alert: CertificateExpired", details: [], metadata: MetadataMap { headers: {} } error.sources=[controller linkerd-identity-headless.linkerd.svc.cluster.local:8080: endpoint 10.200.39.163:8080: connection error: received fatal alert: CertificateExpired, endpoint 10.200.39.163:8080: connection error: received fatal alert: CertificateExpired, connection error: received fatal alert: CertificateExpired, received fatal alert: CertificateExpired] [707002.681759s] INFO ThreadId(01) outbound:proxy{addr=10.20.233.112:8080}:service{ns=spr-apps name=live-reporting-ms-tier1-svc port=8080}:endpoint{addr=10.200.5.23:8080}:rescue{client.addr=10.200.49.6:41100}: linkerd_app_core::errors::respond: gRPC request failed error=endpoint 10.200.5.23:8080: connection error: received fatal alert: CertificateExpired error.sources=[connection error: received fatal alert: CertificateExpired, received fatal alert: CertificateExpired]
[706995.441763s] ERROR ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_proxy_identity_client::certify: Failed to obtain identity error=status: Unknown, message: “controller linkerd-identity-headless.linkerd.svc.cluster.local:8080: endpoint 10.200.39.163:8080: connection error: received fatal alert: CertificateExpired”, details: , metadata: MetadataMap { headers: {} } error.sources=[controller linkerd-identity-headless.linkerd.svc.cluster.local:8080: endpoint 10.200.39.163:8080: connection error: received fatal alert: CertificateExpired, endpoint 10.200.39.163:8080: connection error: received fatal alert: CertificateExpired, connection error: received fatal alert: CertificateExpired, received fatal alert: CertificateExpired][707002.681759s] INFO ThreadId(01) outbound:proxy{addr=10.20.233.112:8080}:service{ns=namespace name=internal-service-name port=8080}:endpoint{addr=10.200.5.23:8080}:rescue{client.addr=10.200.49.6:41100}: linkerd_app_core::errors::respond: gRPC request failed error=endpoint 10.200.5.23:8080: connection error: received fatal alert: CertificateExpired error.sources=[connection error: received fatal alert: CertificateExpired, received fatal alert: CertificateExpired]
Resolved after same restart sequence as above,
Still not sure why it happened for specific workloads, there was no notable resource throttling for these, can you help in suggesting method/metric to detect/dubug this in future? Thanks.