After upgrading to 2.13.0, one of our services is emitting the following log:
INFO ThreadId(02) inbound:server{port=4143}:rescue{client.addr=<ip>:40656}: linkerd_app_core::errors::respond: Request failed error=no route found for request
The pod receives traffic on port 4444 which happens to be part of the default opaque ports list, so I think we are running into this bug, but we are not seeing this behaviour on 2.12.x so I’m not 100% sure it is the same thing.
I will see if we can move this to a different, non-opaque port and rule it out.
I have created an alert in our logs based alerting system because we need to know when this happens, and on that note, should this really be an Error log rather than Info? This means requests are silently failing.
Hmm, this does sound like you’re hitting issue #9888 (the one you linked to). I note that the inbound server port is 4143, the default proxy inbound listener, implying that the connection was made directly to the proxy’s inbound port. This means that the tagged opaque transport is in use, as in #9888.
But, it’s strange that this isn’t observed on 2.12.x. However, I believe #9888 didn’t effect default opaque ports on 2.12, only ones configured by an annotation. But, it’s possible that the destination controller’s behavior has changed in 2.13 and opaque transport hints are now being returned for those ports, resulting in the outbound proxy attempting to use opaque transport here as well.
Regardless, though, the fix for #9888 should still fix this issue as well, given that it also appears to be due to an inbound proxy attempting to handle a tagged transport connection as HTTP.
I would say this bug has become a higher priority in 2.13.0 with it now affecting default opaque ports, as before users would have to specify opaque so they would at least be somewhat aware to check functionality. In this case, it can happen completely blindly.
What do you think to this log being INFO currently?
Hi @lh_dan, I just wanted to let you know that the fix for #9888 has been released in the latest Linkerd edge release, edge-23.4.2, if you’re interested in trying it out. Once the change has been validated on edge, it should make it back to a stable release in the near future.