I am trying to upgrade linkerd from stable-2.14.10 to enterprise-2.15.5 using the operator. I followed the documentation for migrating to BEL’s lifecycle automation from a Helm installation, and then tried to upgrade the control plane by following the documentation for upgrading BEL using the lifecycle automation operator.
I ran kubectl edit controlplane -n linkerd cp-update and edited spec.components.linkerd.controlPlaneConfig.version to enterprise-2.15.5 and I also populated the .license field. It took an hour until the status switched to UpToDate.
$ kubectl describe controlplane -n linkerd cp-update
# -- snip --
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Pending 64m (x2 over 81m) buoyant-operator control-plane update failed: control-plane not ready: linkerd control-plane configMap version [enterprise-2.15.5] but linkerd/Deployment/metrics-api has proxy version [stable-2.14.10]
Warning Pending 54m buoyant-operator control-plane update failed: control-plane not ready: linkerd control-plane configMap version [enterprise-2.15.5] but linkerd/Deployment/tap has proxy version [stable-2.14.10]
Warning Pending 14m buoyant-operator control-plane update failed: control-plane not ready: linkerd control-plane configMap version [enterprise-2.15.5] but linkerd/Deployment/tap-injector has proxy version [stable-2.14.10]
Normal UpToDate 4m40s buoyant-operator control-plane update to enterprise-2.15.5 succeeded
When I first saw those warnings, I tried to push things along by manually restarting deployments:
kubectl rollout restart -n linkerd deploy
I have two retrospective questions:
- Is it normal for the upgrade to take an hour? My cluster is pretty small, 15 nodes.
- Was it necessary for me to restart the deployments manually? (Were the warnings expected and safe to ignore?)
Hi @jennydaman ! No, that’s definitely not expected., and the restart should not have been necessary either. Are there logs from the operator that provide any useful information? Thanks!
Hey @wfresno sorry for the gap in response, I was out of office.
Unfortunately the logs weren’t kept around, but I am trying to upgrade from enterprise-2.15.5 to enterprise-2.16.0 and observing the same error.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Updating 102m buoyant-operator control-plane started updating to enterprise-2.16.0
Warning Pending 97m buoyant-operator control-plane update failed: control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]
Here are the logs from the operator (note the “chris” namespace stuff are the application we are trying to mesh up):
$ kubectl logs -n linkerd-buoyant deploy/linkerd-control-plane-operator
2024/08/25 21:24:21 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota
time="2024-08-25T21:24:22Z" level=info msg="running version v0.32.0"
time="2024-08-25T21:24:22Z" level=info msg="waiting for caches to sync" client=k8s
time="2024-08-25T21:24:22Z" level=info msg="waiting for caches to sync"
time="2024-08-25T21:24:22Z" level=info msg="caches synced"
time="2024-08-25T21:24:22Z" level=info msg="caches synced" client=k8s
time="2024-08-25T21:24:22Z" level=info msg="waiting for caches to sync"
time="2024-08-25T21:24:22Z" level=info msg="caches synced"
time="2024-08-25T21:24:22Z" level=info msg=starting operator=control-plane
time="2024-08-27T18:42:55Z" level=warning msg="error parsing proxy version [stable-2.14.10] for chris-staging/chris1222-postgresql-0: versions stable-2.14.10 and enterprise-2.16.0 are not on the same channel" operator=control-plane
time="2024-08-27T18:42:55Z" level=warning msg="error parsing proxy version [stable-2.14.10] for chris-staging/chris1222-pfdcm-69c4fdbd7c-qc8t7: versions stable-2.14.10 and enterprise-2.16.0 are not on the same channel" operator=control-plane
time="2024-08-27T18:42:55Z" level=warning msg="error parsing proxy version [stable-2.14.10] for chris/pfdcm-linkerd-workaround-linkerd-nodeport-workaround-64dffpsbgd: versions stable-2.14.10 and enterprise-2.16.0 are not on the same channel" operator=control-plane
time="2024-08-27T18:42:55Z" level=warning msg="error parsing proxy version [stable-2.14.10] for chris/chris2222-server-d5c6bf799-qnfj6: versions stable-2.14.10 and enterprise-2.16.0 are not on the same channel" operator=control-plane
time="2024-08-27T18:42:55Z" level=warning msg="error parsing proxy version [stable-2.14.10] for chris/chris2222-server-d5c6bf799-lnnzv: versions stable-2.14.10 and enterprise-2.16.0 are not on the same channel" operator=control-plane
time="2024-08-27T18:42:55Z" level=info msg="linkerd-crds: installed version enterprise-2.15.5 - target version enterprise-2.16.0 - initiating upgrade" operator=control-plane
time="2024-08-27T18:42:55Z" level=info msg=Upgrading... chart=linkerd-crds operator=control-plane release=linkerd-crds version=enterprise-2.16.0
time="2024-08-27T18:43:00Z" level=info msg="successfully upgraded" chart=linkerd-crds operator=control-plane release=linkerd-crds version=enterprise-2.16.0
time="2024-08-27T18:43:00Z" level=info msg="linkerd-control-plane: installed version enterprise-2.15.5 - target version enterprise-2.16.0 - initiating upgrade" operator=control-plane
time="2024-08-27T18:43:00Z" level=info msg="using overrides provided by the ControlPlaneConfig field" operator=control-plane
time="2024-08-27T18:43:00Z" level=info msg=Upgrading... chart=linkerd-control-plane operator=control-plane release=linkerd-control-plane version=enterprise-2.16.0
time="2024-08-27T18:43:17Z" level=info msg="successfully upgraded" chart=linkerd-control-plane operator=control-plane release=linkerd-control-plane version=enterprise-2.16.0
time="2024-08-27T18:43:17Z" level=info msg="Checking control-plane readiness" operator=control-plane
time="2024-08-27T18:48:17Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T18:48:19Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T18:48:19Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T18:48:21Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T18:48:21Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T18:48:25Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T18:48:25Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T18:48:33Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T18:48:33Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T18:48:49Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T18:48:49Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T18:49:21Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T18:49:21Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T18:50:25Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T18:50:25Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T18:52:33Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T18:52:33Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T18:56:49Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T18:56:49Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T19:05:21Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T19:05:21Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T19:15:21Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T19:15:21Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T19:25:21Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T19:25:21Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T19:35:21Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T19:35:21Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T19:45:21Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T19:45:21Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T19:55:21Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T19:55:21Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T20:05:21Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T20:05:21Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T20:15:21Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T20:15:21Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"
time="2024-08-27T20:25:21Z" level=info msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]" operator=control-plane
time="2024-08-27T20:25:21Z" level=error msg="control-plane not ready: linkerd control-plane configMap version [enterprise-2.16.0] but linkerd/Deployment/metrics-api has proxy version [enterprise-2.15.5]"