Hi everyone. I am trying to remove BEL from my Kubernetes cluster. However, although the uninstall commands complete successfully, I still see linkerd resources (pods, deployments, services, control plane operator) running in my cluster. I expected that these things, including the linkerd namespace, would be deleted.
BEL had been installed with the lifecycle automation operator:
NAME STATUS DESIRED CURRENT AGE
linkerd-control-plane Failed enterprise-2.16.1 enterprise-2.15.4 168d
(It’s because this update failed that I decided to uninstall and reinstall.)
$ linkerd version
Client version: enterprise-2.16.1
Server version: enterprise-2.15.4
I manually removed the injection annotation from each affected workload, and restarted them. Next:
$ linkerd viz uninstall | kubectl delete -f -
the server could not find the requested resource (get servers.policy.linkerd.io)
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-viz-metrics-api" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-viz-prometheus" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-viz-tap" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-viz-tap-admin" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-viz-web-api" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-viz-web-check" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-tap-injector" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-metrics-api" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-prometheus" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-tap" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-tap-auth-delegator" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-web-admin" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-web-api" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-web-check" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-tap-injector" deleted
role.rbac.authorization.k8s.io "web" deleted
rolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-tap-auth-reader" deleted
rolebinding.rbac.authorization.k8s.io "web" deleted
apiservice.apiregistration.k8s.io "v1alpha1.tap.linkerd.io" deleted
mutatingwebhookconfiguration.admissionregistration.k8s.io "linkerd-tap-injector-webhook-config" deleted
namespace "linkerd-viz" deleted
authorizationpolicy.policy.linkerd.io "metrics-api" deleted
authorizationpolicy.policy.linkerd.io "prometheus-admin" deleted
authorizationpolicy.policy.linkerd.io "tap" deleted
authorizationpolicy.policy.linkerd.io "tap-injector" deleted
Then
$ linkerd uninstall | kubectl delete -f -
clusterrole.rbac.authorization.k8s.io "linkerd-heartbeat" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-autoregistration" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-autoregistration-gc" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-controller" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-destination" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-enterprise" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-identity" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-proxy-injector" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-sp-validator" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-policy" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-destination-policy" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-heartbeat" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-autoregistration" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-autoregistration-gc" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-controller" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-destination" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-enterprise" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-identity" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-proxy-injector" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-sp-validator" deleted
role.rbac.authorization.k8s.io "linkerd-heartbeat" deleted
role.rbac.authorization.k8s.io "remote-discovery" deleted
rolebinding.rbac.authorization.k8s.io "linkerd-destination-remote-discovery" deleted
rolebinding.rbac.authorization.k8s.io "linkerd-heartbeat" deleted
customresourcedefinition.apiextensions.k8s.io "authorizationpolicies.policy.linkerd.io" deleted
customresourcedefinition.apiextensions.k8s.io "externalworkloads.workload.linkerd.io" deleted
customresourcedefinition.apiextensions.k8s.io "httproutes.gateway.networking.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "httproutes.policy.linkerd.io" deleted
customresourcedefinition.apiextensions.k8s.io "meshtlsauthentications.policy.linkerd.io" deleted
customresourcedefinition.apiextensions.k8s.io "networkauthentications.policy.linkerd.io" deleted
customresourcedefinition.apiextensions.k8s.io "serverauthorizations.policy.linkerd.io" deleted
customresourcedefinition.apiextensions.k8s.io "servers.policy.linkerd.io" deleted
customresourcedefinition.apiextensions.k8s.io "serviceprofiles.linkerd.io" deleted
mutatingwebhookconfiguration.admissionregistration.k8s.io "linkerd-proxy-injector-webhook-config" deleted
validatingwebhookconfiguration.admissionregistration.k8s.io "linkerd-policy-validator-webhook-config" deleted
validatingwebhookconfiguration.admissionregistration.k8s.io "linkerd-sp-validator-webhook-config" deleted
However, despite that, the namespace and many things in it still exist and are running:
$ kubectl get po -n linkerd
NAME READY STATUS RESTARTS AGE
linkerd-destination-5796cf9798-2jpkz 4/4 Running 0 129d
linkerd-destination-5796cf9798-nr9nw 4/4 Running 0 69d
linkerd-destination-5796cf9798-xvhrk 4/4 Running 0 129d
linkerd-identity-66fff5c886-5xbfq 2/2 Running 0 69d
linkerd-identity-66fff5c886-7tt7n 2/2 Running 0 129d
linkerd-identity-66fff5c886-r4954 2/2 Running 0 129d
linkerd-proxy-injector-5bf54847b9-6md7k 2/2 Running 0 129d
linkerd-proxy-injector-5bf54847b9-hht2n 2/2 Running 0 129d
linkerd-proxy-injector-5bf54847b9-kh7pr 2/2 Running 0 69d
$ kubectl get deploy -n linkerd
NAME READY UP-TO-DATE AVAILABLE AGE
linkerd-destination 3/3 3 3 168d
linkerd-identity 3/3 3 3 168d
linkerd-proxy-injector 3/3 3 3 168d
$ kubectl get svc -n linkerd
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
linkerd-dst ClusterIP 10.0.6.166 <none> 8086/TCP 168d
linkerd-dst-headless ClusterIP None <none> 8086/TCP 168d
linkerd-identity ClusterIP 10.0.253.166 <none> 8080/TCP 168d
linkerd-identity-headless ClusterIP None <none> 8080/TCP 168d
linkerd-policy ClusterIP None <none> 8090/TCP 168d
linkerd-policy-validator ClusterIP 10.0.33.27 <none> 443/TCP 168d
linkerd-proxy-injector ClusterIP 10.0.71.101 <none> 443/TCP 168d
linkerd-sp-validator ClusterIP 10.0.75.100 <none> 443/TCP 168d
Thanks for any advice!
Ben