HTTP Requests Over single GRPC channel are not being distributed among pods in replica set

Hello. We are trying to use Linkerd for a problem which I believe is very common - GRPC channels do not work well with Kubernetes’ native load balancing. I have a Kubernetes service (type LoadBalancer) which routes to a replica set of pods. I have one client which forms a handful of long-standing grpc connections to the load balancer, and sends high latency (~3s) requests a few times a second.
My issue is that the linkerd layer 7 routing does not appear to be working. All traffic coming from a single connection still gets routed to a single pod. Here is an example of route statistics:

According to Link , all installation and setup is correct, linkerd checklooks fine, and the linkerd viz shows correct information.
Any help would be appreciate, thank you.

Hey Joe!

In order for Linkerd to take over routing you need to send your request to the clusterIP, not the external loadbalancer. Update the client to talk to the cluster IP and you should see the behavior you expect.

Thanks,
Jason

Hi Jason, thank you for the response. I’ve modified my service manifest to type: ClusterIP and directed my client towards that address, but unfortunately the statistics still seem the same.

A piece of information more specific to my situation is that I route traffic to the service through a vpn - hosted by a pod within the cluster. However I’m not certain that would affect anything. To confirm, I had the client open a new grpc connection each request, which showed route statistics which I would like to replicate with a single connection:

Note that these are dummy requests - which is why the latencies are low.