# How do I know which endpoints Linkerd is aware of? (was: Fail-Fast troubleshooting) **URL:** \ **Category:** Linkerd General Discussion\ **Created:** [April 11, 2023, 10:53am UTC](https://linkerd.buoyant.io/t/how-do-i-know-which-endpoints-linkerd-is-aware-of-was-fail-fast-troubleshooting/74 "2023-04-11T10:53:33Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Eli-Goldberg](https://avatars.discourse-cdn.com/v4/letter/e/919ad9/32.png) [@Eli-Goldberg](https://linkerd.buoyant.io/u/Eli-Goldberg)\ **Post date:** [April 11, 2023, 10:53am UTC](https://linkerd.buoyant.io/t/how-do-i-know-which-endpoints-linkerd-is-aware-of-was-fail-fast-troubleshooting/74/1 "2023-04-11T10:53:33Z")
Hi all ! I’m trying to figure out the reason for a “fail-fast” error that occurred between 2 meshed services (A is the client, B is the server). Is there a way to know at a given point which endpoints service A is currently aware of? To the best of my understanding, “fail-fast” would happen when a client says: “I don’t know of any endpoints to send requests to, so I’m done”. In this case Service B had 3 instances but one of them crashed (OOM) - which triggered one of the 2 instances of Service A to “fail-fast”. I suspect Service A’s endpoints were not updated. Perhaps this has to do max-concurrent-requests too, like in this issue: > > > \### What is the issue? > > The Linkerd proxy on a random \[Cortex\](https://cortexmet…rics.io/) Ingester enters into the fail-fast mode blocking the communication against the distributors, but not against other Cortex components like Queriers. > > That effectively breaks replication as the distributors cannot see the Ingester in a healthy state, even if the communication via \`memberlist\` is unaffected and the Ingester appears active on the ring. > > I tried restarting the Ingester, but the problem solves temporarily. The strange part is that, sometimes, another Ingester enters into the fail-fast state after restarting the affected one, which is why I used the term random to describe the problem. > > > \### How can it be reproduced? > > The problem appears when handling a considerable amount of traffic. Currently, distributors are receiving a constant rate of 50K samples per second in batches of 500, meaning, effectively, distributors are receiving 100 requests (with 500 samples each), and according to \`linkerd viz dashboard\`, Ingesters are receiving a similar number of RPS. > > On my initial tests with orders of magnitudes less traffic, the problem doesn't appear. > > \### Logs, error output, etc > > As the logs are very verbose due to the injection rate. Here are the last 5 seconds from the affected Ingester (i.e., \`ingester-0\`) and the two distributors: > > https://gist.github.com/agalue/5ecbbfcf37ecf8b5798bf18bbe0473b1 > > Here is how I got the logs: > > \`\`\`bash > kubectl logs -n cortex distributor-ddd56cf9-4sz4s -c linkerd-proxy --since=5s \> distributor-4sz4s-proxy-logs.txt > kubectl logs -n cortex distributor-ddd56cf9-wzjdd -c linkerd-proxy --since=5s \> distributor-wzjdd-proxy-logs.txt > kubectl logs -n cortex ingester-0 linkerd-proxy --since=5s \> ingester-0-proxy-logs.txt > \`\`\` > > \### output of \`linkerd check -o short\` > > \`\`\`bash > ➜ ~ linkerd check -o short > Status check results are √ > \`\`\` > > \### Environment > > \- Kubernetes Version: 1.23.3 > \- Cluster Environment: AKS with Kubenet and Calico > \- Host OS: Ubuntu 18.04.6 LTS, 5.4.0-1083-azure, containerd://1.5.11+azure-2 (managed by AKS) > \- Linkerd Version: 2.11.4 > > Note: the problem appears with and without Calico (tested on different clusters). > > > \### Possible solution > > \_No response\_ > > \### Additional context > > In Cortex, all components talk to each other via Pod IP, meaning all the communication happens via Pod-to-Pod through gRPC. > > To give more context about what you would see on the proxy logs: > > \`\`\`bash > ➜ ~ kubectl get pod -n cortex -o wide > NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES > compactor-0 2/2 Running 0 47h 10.244.24.10 aks-rpsupport-13025066-vmss000004 \ \ > distributor-ddd56cf9-4sz4s 2/2 Running 0 46h 10.244.2.26 aks-distributors-15878912-vmss000000 \ \ > distributor-ddd56cf9-wzjdd 2/2 Running 0 46h 10.244.28.8 aks-distributors-15878912-vmss00000d \ \ > ingester-0 2/2 Running 0 24h 10.244.12.14 aks-ingesters-39599960-vmss000001 \ \ > ingester-1 2/2 Running 0 47h 10.244.4.13 aks-ingesters-39599960-vmss000000 \ \ > ingester-2 2/2 Running 0 43h 10.244.3.17 aks-ingesters-39599960-vmss000002 \ \ > memcached-chunks-0 3/3 Running 0 47h 10.244.8.26 aks-rpsupport-13025066-vmss000001 \ \ > memcached-chunks-1 3/3 Running 0 47h 10.244.10.28 aks-rpsupport-13025066-vmss000002 \ \ > memcached-chunks-2 3/3 Running 0 47h 10.244.6.33 aks-rpsupport-13025066-vmss000003 \ \ > memcached-frontend-0 3/3 Running 0 47h 10.244.10.30 aks-rpsupport-13025066-vmss000002 \ \ > memcached-frontend-1 3/3 Running 0 47h 10.244.8.25 aks-rpsupport-13025066-vmss000001 \ \ > memcached-frontend-2 3/3 Running 0 47h 10.244.6.32 aks-rpsupport-13025066-vmss000003 \ \ > memcached-index-0 3/3 Running 0 47h 10.244.10.29 aks-rpsupport-13025066-vmss000002 \ \ > memcached-index-1 3/3 Running 0 47h 10.244.8.24 aks-rpsupport-13025066-vmss000001 \ \ > memcached-index-2 3/3 Running 0 47h 10.244.6.31 aks-rpsupport-13025066-vmss000003 \ \ > memcached-metadata-0 3/3 Running 0 47h 10.244.6.34 aks-rpsupport-13025066-vmss000003 \ \ > querier-794978b45f-2b7z2 2/2 Running 0 47h 10.244.23.6 aks-storegws-30442145-vmss000013 \ \ > querier-794978b45f-h2fmf 2/2 Running 0 47h 10.244.15.13 aks-storegws-30442145-vmss00000w \ \ > querier-794978b45f-vbjqn 2/2 Running 0 47h 10.244.17.8 aks-storegws-30442145-vmss00000z \ \ > query-frontend-5b57ddb6cf-bkvpk 2/2 Running 0 47h 10.244.6.35 aks-rpsupport-13025066-vmss000003 \ \ > query-frontend-5b57ddb6cf-jxgq2 2/2 Running 0 47h 10.244.8.27 aks-rpsupport-13025066-vmss000001 \ \ > store-gateway-0 2/2 Running 0 47h 10.244.23.5 aks-storegws-30442145-vmss000013 \ \ > store-gateway-1 2/2 Running 0 47h 10.244.17.7 aks-storegws-30442145-vmss00000z \ \ > store-gateway-2 2/2 Running 0 47h 10.244.15.12 aks-storegws-30442145-vmss00000w \ \ > \`\`\` > > The only error code I found on the distributors proxy is: > > \`\`\`text > \[165766.325281s\] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.2.26:49942}: linkerd\_app\_core::serve: Connection closed reason=connection error: server: Transport endpoint is not connected (os error 107) > \`\`\` > > In terms of the applications, the affected Ingester reports nothing on its log, as the distributor traffic is not reaching the application. > > The distributors, on the other hand, are flooded with the following message; as I presume the proxy on the affected Ingester is rejecting the traffic: > > \`\`\`text > level=warn ts=2022-07-18T15:57:05.907166697Z caller=pool.go:184 msg="removing ingester failing healthcheck" addr=10.244.3.14:9095 reason="rpc error: code = Unavailable desc = HTTP Logical service in fail-fast" > \`\`\` > > > \### Would you like to work on fixing this bug? > > \_No response\_ ---
**Author:** ![jmo](https://yyz1.discourse-cdn.com/flex029/user_avatar/linkerd.buoyant.io/jmo/32/62_2.png) [@jmo](https://linkerd.buoyant.io/u/jmo)\ **Post date:** [April 11, 2023, 2:48pm UTC](https://linkerd.buoyant.io/t/how-do-i-know-which-endpoints-linkerd-is-aware-of-was-fail-fast-troubleshooting/74/2 "2023-04-11T14:48:23Z")
Hey Eli! The `linkerd diagnostics endpoints` command will list out the endpoints for a given service. A quick example using the emojivoto app would look like: `linkerd diagnostics endpoints emoji-svc.emojivoto.svc.cluster.local:8080 web-svc.emojivoto.svc.cluster.local:80`