We run on GKE and we’ve seen two instances in the past month where a GKE node will get removed, and immediately replaced by another GKE node, and the new node is given the same pod CIDR as the deleted node. Additionally, we see that daemonsets on the deleted node remain in the kube API for some time after the node is deleted. This results in a brief period of time where there are multiple “Running” pods in the kube API with the same IP address, which causes errors like the following, when clients attempt to dial the “new” pods which have been given the same IP address as daemonset pods on the deleted node:
Failed to subscribe to profile by ip {POD_IP_ADDR}: rpc error: code = FailedPrecondition desc = found 2 pods with a conflicting network IP {POD_IP_ADDR}
We received this explanation from GKE support, on the issue:
-When a node scale-down occurs, GKE cordons and drains the node. By default, the drain process ignores DaemonSet workloads, leaving them registered on the node.
-When the underlying GCE instance is forcefully deleted, the kubelet process is terminated immediately. Because the DaemonSet pods were not cleanly evicted prior to this, kubelet is unable to notify the Kubernetes API server that the pods have terminated. The API server continues to reflect them in the Running state.
-To prevent subnet IP address exhaustion, GKE's IPAM controller immediately reclaims and assigns the released Pod CIDR block to the newly created instance.
-The Kubernetes cloud-controller-manager asynchronously processes VM terminations and purges orphaned Pod objects via the pod-garbage-collector (PodGC). This introduces a temporary synchronization gap (~1-2 minutes), during which a newly scheduled pod on the replacement node can receive the same IP address before the API Server has garbage-collected the old pod.
We use gRPC, and pods dial other pods directly by their pod IP. When we hit this scenario, clients hit this “conflicting IP” error on the “GetProfile” request and, when this happens, all requests to the new pods fail because they are rejected by our linkerd policies.
Additionally, even after the old daemonset pods are deleted, and there is no longer an IP overlap, requests to the new pods continue to fail and be rejected by our linkerd policies. What I’m unsure about is whether this is because client proxies don’t retry the GetProfile request, or whether it’s because gRPC is using long-lived connections and, once the initial connection is made, it’s in a bad state from which it does not self-recover (even if the client proxy subsequently makes a successful GetProfile) request.
I found a fairly old github issue which seems very similar, because it mentions pod IP issues with GKE and daemonsets, but it seems the issue was never fully root caused at the time: linkerd-destination seemingly serving incorrect Pod IPs causing connection errors · Issue #8956 · linkerd/linkerd2 · GitHub
I had two specific questions, to help us mitigate this issue.:
- Does the linkerd client-proxy retry the GetProfile request when it encounters this FailedPrecondition error? I’m wondering whether the reason we continue to see the failed requests after the old daemonset pods are deleted is because the GetProfile request is not retried, or whether it’s because the initial gRPC connection is long-lived.
- Our gRPC libraries currently use direct pod IP addressing. We’re evaluating moving our gRPC client-side load-balancing onto linkerd, so that we use k8s service addressing and let linkerd manage the pod connections. My understanding is that we currently encounter this issue because we do direct pod IP addressing, and thus the GetProfile request is made with the pod IP address, which errors when the destination service sees multiple pods with the same IP address. If we move to addressing by k8s service, I assume that the GetProfile request will be made with the k8s service DNS name, and this issue “conflicting IP” error will no longer be an issue in this scenario, is that correct?
Any assistance with this is greatly appreciated, thanks!