We are observing intermittent readiness and liveness probe failures for Linkerd pods (specially Linkerd-destination pod)of the gke cluster. These failures result in pod restarts due to liveness probe timeouts.The Kubernetes version on which it is running is 1.31.
Error message:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 41m (x41421 over 103d) kubelet Liveness probe failed: Get "http://10.99.78.5:9996/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 31m (x42777 over 103d) kubelet Readiness probe failed: Get "http://10.99.78.5:9996/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 26m (x11367 over 103d) kubelet (combined from similar events): Readiness probe failed: Get "http://10.99.78.5:9997/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 21m (x41678 over 103d) kubelet Readiness probe failed: Get "http://10.99.78.5:9990/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 11m (x41286 over 103d) kubelet Liveness probe failed: Get "http://10.99.78.5:9997/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 6m44s (x41542 over 103d) kubelet Liveness probe failed: Get "http://10.99.78.5:9990/live": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 64s (x40672 over 103d) kubelet Readiness probe failed: Get "http://10.99.78.5:9997/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
I’d like to get to the bottom of this to understand what could be causing this issue.
