Many warning logs in Linkerd Destination container since upgrading to v2.13.x

Hi all,

We’re in the proces of upgrading Linkerd from v2.12.4 to v2.13.5 in our clusters. Everything seems to be working fine, but we’re seeing a lot of warning logs in the destination container which were not there before. We haven’t changed any configuration.

I’ve tried searching for the specific warning that is being logged, but I couldn’t find it anywhere in the docs, issues, etc.

The warning(s) are:

time="2023-07-11T12:22:28Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:xxx namespace:yyy port:zzz]" addr=":8086" component=server

Is there anyone who might be able to clarify what this warning is trying to tell us?

Thanks!

Looking a bit further, I see that a fix for this bug was merged a while ago (Fix `server_port_subscribers` metric in Destination by alpeb · Pull Request #10820 · linkerd/linkerd2 · GitHub).

Looking at this comment stating that it has been an issue since v2.13.4, could it be that this issue is reintroduced recently?

Based on reports of this issue still being present in 2.13.5, we have reopened unable to delete server_port_subscribers metric with labels · Issue #10764 · linkerd/linkerd2 · GitHub

It would be really helpful if you could share destination controller logs in a comment on that issue. This will let us see the exact text of the error and help give us the context to reproduce and ultimately fix the error. Thanks!

Hi, I’m also seeing a lot of this error in the destination logs:

Setup:
linkerd version: stable-2.13.5
HA mode
CNI enabled
external prometheus

We’re currently working on a fix, that’ll be included in 2.13.6 to be released very soon. Note however this is an innocuous warning that affects the server_port_subscribers metric, which is just an informative metric, and doesn’t affect the operation of the controller.

Unfortunately, the fix didn’t make it into an edge yet, so we couldn’t include it in 2.13.6 released yesterday. I just want to reiterate that these warnings can be disregarded.

While it didn’t make it into 2.13.6, this fix is, I believe, in 2.14.1 and later.