Breaking change in Linkerd 2.16.1 proxy. Shutdown endpoint is missing

Linkerd General Discussion
1

In some pods we use this command to shut down the proxy after a main container is completed.
trap “wget -O - -q --post-data shutdown http://127.0.0.1:4191/shutdown” EXIT

It as worked for all versions of Linkerd, but has stopped working when we upgraded from version 2.15 to 2.16.1. This breaks our clusters as some pods are not terminated anymore.

Now the above command just fails with: wget: server returned error: HTTP/1.1 404 Not Found

I did not see any release notes on this being a breaking change. We need a fix for this to be able to upgrade to version 2.16

2

HI Jan, this was a change in 2.16.0. Please see the 2.16.0 release notes especially the section marked “Breaking changes to the shutdown endpoint”.

3

Ah. I see it now. And with a workaround to enable it again until we are running in sidecar mode everywhere. I’ve been waiting for some information that sidecars are no longer experimental in Linkerd. Is it safe to run with it in production for k8s 1.29 or newer?

4

Yes, it’s safe. We have several customers who are using 2.16.x and native sidecars in production on K8s 1.29+. It makes things a lot nicer especially for Jobs. Let us know if you run into any issues.

5

Thanks for the information. I’ll start rolling it out on our production clusters

6

@william I see there is one minor issue with running in sidecar mode. We use ArgoCD for deployment of Linkerd. When enabling sidecars we always end up with a diff in the controlplane helm chart after sync. It happens because you specify the startupProbe.initialDelaySeconds even if it has the default value. This means that the desired workload yaml has this value specified and Kubernetes will always remove it since it has the default value. Leaving the deployments in the controlplane as OutOfSync. The workaround is to add a ignoreDifferences in ArgoCD. But it I don’t really like to do that. It would be better if you do not specify the property with the default value.
This is the workaround to fix it in ArgoCD:

      ignoreDifferences:
        - group: apps
          kind: Deployment
          name: linkerd-proxy-injector
          jqPathExpressions:
            - '.spec.template.spec.initContainers[]?.startupProbe?.initialDelaySeconds'
        - group: apps
          kind: Deployment
          name: linkerd-destination
          jqPathExpressions:
            - '.spec.template.spec.initContainers[]?.startupProbe?.initialDelaySeconds'
        - group: apps
          kind: Deployment
          name: linkerd-enterprise
          jqPathExpressions:
            - '.spec.template.spec.initContainers[]?.startupProbe?.initialDelaySeconds'