We recently upgraded some of our environments from Linkerd 2.14.1 to 2.14.2. After that we are experiencing issues where meshed pods cannot connect to unmeshed smtp/redis pods. The connection just hangs until the client side timeout. This is typically connections to smtp or redis statefulsets. I have not seen issues for http. This is causing a lot of problems and instability.
The connection failure is not a 100%. It seems like after the system gets under a certain load or number of pods grow this state occurs and it remains that way. We can see that in a low traffic period the connection may succeed and then later fail if the client side pod needs to reconnect.
Workarounds:
If I add the smtp and redis statefulset pods to the service mesh then connections succeed again.
Also if I downgrade the control plane to 2.14.1 then the connections immediately succeed
For now we have to downgrade to Linkerd 2.14.1 since we in some of our environments have smtp and redis running unmeshed. And we also do not know what other impact this bug will have
Let me know if you need more information to be able to fix this bug. Looking forward to upgrade to 2.14.3 with a fix for this issue
This is a little surprising! Thanks for letting us know about this.
In order for us to identify how the behavior regressed, perhaps you can help us fill in some details:
The target workload is a statefulset. How do clients connect to these pods? Do clients resolve a DNS name in the form of $svc.$ns.cluster.svc.local? If so is this service a Headless service (i.e. with clusterIP: None)? Or are you using stable network IDs to resolve specific endpoints?
Do these statefulset pods expose ports via HostPort? Or is Host networking enabled on these pods?
We can see that in a low traffic period the connection may succeed and then later fail if the client side pod needs to reconnect.
To confirm: there are no changes (i.e. rollouts) of the statefulset during this time period? The statefulset remains unchanged and clients appear to stop being able to connect to it?
Thanks for narrowing this down to headless, unmeshed services. We can start to work on a repro. Do you happen to have the proxy logs showing which IPs the client was trying to connect to when it timed out? And how that compared to the status of these headless services (and underlying pods) in the Kubernetes API?
I have some logs. But not sure how useful they are. 10.241.28.203 is the IP address of one of the redis pods. I did the rollback of the Linkerd control plane to 2.14.1 around 07:52. At that time the client pods connected immediately.
When looking for the logs I also looked at one of the Linkerd destination pod logs. This log is after the rollback to 2.14.1. It has some errors there about the SMTP Statefulset. The smtp statefulset connection works after the rollback, but I still don’t like the look of those error messages. See log et the end of this post
2023-10-31T04:50:27.592282726Z [ 0.002531s] INFO ThreadId(01) linkerd2_proxy: release 2.210.1 (dd47e38) by linkerd on 2023-10-26T18:25:56Z
2023-10-31T04:50:27.605655136Z [ 0.011603s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
2023-10-31T04:50:27.605670537Z [ 0.012456s] INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
2023-10-31T04:50:27.605675537Z [ 0.012470s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
2023-10-31T04:50:27.605679937Z [ 0.012473s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
2023-10-31T04:50:27.605684437Z [ 0.012476s] INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
2023-10-31T04:50:27.605689237Z [ 0.012479s] INFO ThreadId(01) linkerd2_proxy: Local identity is revelation-backend.horizons.serviceaccount.identity.linkerd.cluster.local
2023-10-31T04:50:27.605695537Z [ 0.012482s] INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
2023-10-31T04:50:27.605700237Z [ 0.012485s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
2023-10-31T04:50:27.634376089Z [ 0.044591s] INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=revelation-backend.horizons.serviceaccount.identity.linkerd.cluster.local
2023-10-31T04:50:39.217838751Z [ 11.624741s] INFO ThreadId(01) inbound:server{port=3000}:rescue{client.addr=10.241.28.104:56124}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T04:50:49.216856473Z [ 21.624645s] INFO ThreadId(01) inbound:server{port=3000}:rescue{client.addr=10.241.28.104:58758}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T04:50:59.208227955Z [ 31.618486s] INFO ThreadId(01) inbound:server{port=3000}:rescue{client.addr=10.241.28.104:58942}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T04:51:09.218979216Z [ 41.629225s] INFO ThreadId(01) inbound:server{port=3000}:rescue{client.addr=10.241.28.104:33626}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T04:51:19.219443618Z [ 51.629681s] INFO ThreadId(01) inbound:server{port=3000}:rescue{client.addr=10.241.28.104:50814}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T04:51:29.209364613Z [ 61.619622s] INFO ThreadId(01) inbound:server{port=3000}:rescue{client.addr=10.241.28.104:60144}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T04:51:44.207255215Z [ 76.617450s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=connection closed before message completed client.addr=10.241.28.104:54546
2023-10-31T04:51:54.207948424Z [ 86.617657s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=connection closed before message completed client.addr=10.241.28.104:57610
2023-10-31T04:52:04.210395884Z [ 96.618954s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=connection closed before message completed client.addr=10.241.28.104:41160
2023-10-31T04:52:14.207175733Z [ 106.617398s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=connection closed before message completed client.addr=10.241.28.104:49228
2023-10-31T04:52:24.206835480Z [ 116.617064s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=connection closed before message completed client.addr=10.241.28.104:40282
2023-10-31T04:52:34.210195300Z [ 126.620483s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=connection closed before message completed client.addr=10.241.28.104:48082
2023-10-31T07:51:26.641074873Z [ 10859.051403s] WARN ThreadId(01) outbound:proxy{addr=10.241.17.138:5432}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.241.28.21:8090}: linkerd_reconnect: Service failed error=endpoint 10.241.28.21:8090: channel closed error.sources=[channel closed]
2023-10-31T07:51:26.669834132Z [ 10859.080129s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:6379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.28.21:8086}: linkerd_reconnect: Service failed error=endpoint 10.241.28.21:8086: channel closed error.sources=[channel closed]
2023-10-31T07:51:27.016772470Z [ 10859.427007s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:6379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.28.21:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.241.28.21:8086: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:28.436688225Z [ 10860.846957s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:6379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.28.21:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.241.28.21:8086: connect timed out after 1s error.sources=[connect timed out after 1s]
2023-10-31T07:51:29.944111600Z [ 10862.354391s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:6379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.28.21:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.241.28.21:8086: connect timed out after 1s error.sources=[connect timed out after 1s]
2023-10-31T07:51:57.206909965Z [ 10889.617039s] WARN ThreadId(01) watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.241.27.155:8090}: linkerd_reconnect: Service failed error=endpoint 10.241.27.155:8090: channel closed error.sources=[channel closed]
2023-10-31T07:51:57.251125606Z [ 10889.654450s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:26379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.27.155:8086}: linkerd_reconnect: Service failed error=endpoint 10.241.27.155:8086: channel closed error.sources=[channel closed]
2023-10-31T07:51:57.594375214Z [ 10890.004663s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:26379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.24.149:8086}: linkerd_reconnect: Service failed error=endpoint 10.241.24.149:8086: channel closed error.sources=[channel closed]
2023-10-31T07:51:57.595230424Z [ 10890.005536s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:26379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.27.155:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.241.27.155:8086: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:57.692754020Z [ 10890.103085s] WARN ThreadId(01) outbound:proxy{addr=10.241.17.138:5432}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.241.24.149:8090}: linkerd_reconnect: Service failed error=endpoint 10.241.24.149:8090: channel closed error.sources=[channel closed]
2023-10-31T07:51:57.693358727Z [ 10890.103695s] WARN ThreadId(01) watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.241.27.155:8090}: linkerd_reconnect: Failed to connect error=endpoint 10.241.27.155:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:57.700382913Z [ 10890.110689s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:26379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.24.149:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.241.24.149:8086: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:57.800402239Z [ 10890.210691s] WARN ThreadId(01) outbound:proxy{addr=10.241.17.138:5432}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.241.24.149:8090}: linkerd_reconnect: Failed to connect error=endpoint 10.241.24.149:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:57.814677314Z [ 10890.225008s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:26379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.27.155:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.241.27.155:8086: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:57.904743118Z [ 10890.315065s] WARN ThreadId(01) watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.241.27.155:8090}: linkerd_reconnect: Failed to connect error=endpoint 10.241.27.155:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:57.910431788Z [ 10890.320752s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:26379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.24.149:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.241.24.149:8086: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:58.002537417Z [ 10890.412812s] WARN ThreadId(01) outbound:proxy{addr=10.241.17.138:5432}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.241.24.149:8090}: linkerd_reconnect: Failed to connect error=endpoint 10.241.24.149:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:58.229357397Z [ 10890.639662s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:26379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.27.155:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.241.27.155:8086: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:58.334273183Z [ 10890.744583s] WARN ThreadId(01) watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.241.27.155:8090}: linkerd_reconnect: Failed to connect error=endpoint 10.241.27.155:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:58.348555558Z [ 10890.758855s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:26379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.24.149:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.241.24.149:8086: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:58.414715369Z [ 10890.825013s] WARN ThreadId(01) outbound:proxy{addr=10.241.17.138:5432}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.241.24.149:8090}: linkerd_reconnect: Failed to connect error=endpoint 10.241.24.149:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:58.850592912Z [ 10891.260902s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:26379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.24.149:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.241.24.149:8086: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:58.916779324Z [ 10891.327085s] WARN ThreadId(01) outbound:proxy{addr=10.241.17.138:5432}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.241.24.149:8090}: linkerd_reconnect: Failed to connect error=endpoint 10.241.24.149:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:59.353616878Z [ 10891.763888s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:26379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.24.149:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.241.24.149:8086: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:59.418943679Z [ 10891.829255s] WARN ThreadId(01) outbound:proxy{addr=10.241.17.138:5432}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.241.24.149:8090}: linkerd_reconnect: Failed to connect error=endpoint 10.241.24.149:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2023-10-31T07:51:59.730572699Z [ 10892.140882s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:26379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.27.155:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.241.27.155:8086: connect timed out after 1s error.sources=[connect timed out after 1s]
2023-10-31T07:51:59.835845789Z [ 10892.246160s] WARN ThreadId(01) watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.241.27.155:8090}: linkerd_reconnect: Failed to connect error=endpoint 10.241.27.155:8090: connect timed out after 1s error.sources=[connect timed out after 1s]
2023-10-31T07:52:00.597152721Z [ 10893.007358s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:26379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_stack::failfast: Service entering failfast after 3s
2023-10-31T07:52:00.861204917Z [ 10893.266637s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:26379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.24.149:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.241.24.149:8086: connect timed out after 1s error.sources=[connect timed out after 1s]
2023-10-31T07:52:01.233307854Z [ 10893.643418s] WARN ThreadId(01) outbound:proxy{addr=10.241.28.203:26379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.241.27.155:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.241.27.155:8086: connect timed out after 1s error.sources=[connect timed out after 1s]
2023-10-31T07:52:01.463308396Z [ 10893.871872s] INFO ThreadId(01) linkerd_stack::failfast: Service has recovered
Linkerd destination pod log after rollback
time=“2023-10-31T07:51:59Z” level=info msg=“running version stable-2.14.1”
time=“2023-10-31T07:51:59Z” level=info msg=“starting admin server on :9996”
time=“2023-10-31T07:51:59Z” level=info msg=“Using default opaque ports: map[25:{} 587:{} 3306:{} 4444:{} 5432:{} 6379:{} 9300:{} 11211:{}]”
time=“2023-10-31T07:51:59Z” level=info msg=“waiting for caches to sync”
time=“2023-10-31T07:52:00Z” level=info msg=“caches synced”
time=“2023-10-31T07:52:00Z” level=info msg=“waiting for caches to sync”
time=“2023-10-31T07:52:00Z” level=info msg=“caches synced”
time=“2023-10-31T07:52:00Z” level=info msg=“waiting for caches to sync”
time=“2023-10-31T07:52:00Z” level=info msg=“caches synced”
time=“2023-10-31T07:52:00Z” level=info msg=“starting gRPC server on :8086”
time=“2023-10-31T08:11:23Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-1
time=“2023-10-31T08:11:35Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-1
time=“2023-10-31T08:11:51Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-0
time=“2023-10-31T08:11:52Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-1
time=“2023-10-31T08:12:08Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-0
time=“2023-10-31T08:12:38Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-0
time=“2023-10-31T08:13:03Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-0
time=“2023-10-31T08:13:04Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-1
time=“2023-10-31T08:19:43Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-1
time=“2023-10-31T08:19:54Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-1
time=“2023-10-31T08:20:05Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-1
time=“2023-10-31T08:20:18Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-0
time=“2023-10-31T08:20:24Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-1
time=“2023-10-31T08:20:53Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-0
time=“2023-10-31T08:20:54Z” level=error msg=“failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec” addr=“:8086” component=server ns=horizons pod=horizons-smtp-0
Stream closed EOF for linkerd/linkerd-destination-667c59776-6l2th (destination)
The fix has been merged and was included in last week’s edge release for testing and validation. We plan to include the fix in the stable 2.14.3 this week.