Proxy Injection Not Working

Linkerd General Discussion
1

Hello everyone,

I am running into an issue injecting a proxy in any deployment. I followed Linkerd Demo App to confirm the problem was not with my own deployment configurations. Any help would be much appreciated!

Emojivoto Web Deployment:

~$ kubectl describe deploy -n emojivoto web
Name:                   web
Namespace:              emojivoto
CreationTimestamp:      Fri, 28 Jun 2024 20:13:45 +0000
Labels:                 app.kubernetes.io/name=web
                        app.kubernetes.io/part-of=emojivoto
                        app.kubernetes.io/version=v11
Annotations:            deployment.kubernetes.io/revision: 2
Selector:               app=web-svc,version=v11
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=web-svc
                    version=v11
  Annotations:      linkerd.io/inject: enabled
  Service Account:  web
  Containers:
   web-svc:
    Image:      docker.l5d.io/buoyantio/emojivoto-web:v11
    Port:       8080/TCP
    Host Port:  0/TCP
    Requests:
      cpu:  100m
    Environment:
      WEB_PORT:        8080
      EMOJISVC_HOST:   emoji-svc.emojivoto:8080
      VOTINGSVC_HOST:  voting-svc.emojivoto:8080
      INDEX_BUNDLE:    dist/index_bundle.js
    Mounts:            <none>
  Volumes:             <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   web-6c5b7bb685 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  61s   deployment-controller  Scaled up replica set web-555b75889b to 1
  Normal  ScalingReplicaSet  46s   deployment-controller  Scaled up replica set web-6c5b7bb685 to 1
  Normal  ScalingReplicaSet  44s   deployment-controller  Scaled down replica set web-555b75889b to 0 from 1

Linkerd Check Emojivoto Proxy:

linkerd-data-plane
------------------
√ data plane namespace exists
DEBU[0003] Retrying on error: no "linkerd-proxy" containers found in the "emojivoto" namespace
/ waiting for check to complete DEBU[0008] Retrying on error: no "linkerd-proxy" containers found in the "emojivoto" namespace
\ waiting for check to complete DEBU[0013] Retrying on error: no "linkerd-proxy" containers found in the "emojivoto" namespace
/ waiting for check to complete DEBU[0018] Retrying on error: no "linkerd-proxy" containers found in the "emojivoto" namespace

Linkerd Proxy Injection Logs:

time="2024-06-28T19:55:55Z" level=info msg="running version edge-24.3.5"
time="2024-06-28T19:55:55Z" level=info msg="starting admin server on :9995"
time="2024-06-28T19:55:55Z" level=info msg="waiting for caches to sync"
time="2024-06-28T19:55:55Z" level=info msg="listening at :8443"
time="2024-06-28T19:55:55Z" level=info msg="caches synced"

List of current running pods in emojivoto:

~$ kubectl get pods -n emojivoto
NAME                        READY   STATUS    RESTARTS   AGE
emoji-79d94b5dcb-nz2nh      1/1     Running   0          9m20s
vote-bot-5dd8699f6f-58p4v   1/1     Running   0          9m20s
voting-99fd4c49f-gvg8g      1/1     Running   0          9m20s
web-6c5b7bb685-pgqg4        1/1     Running   0          9m20s
2

Hi @abarrios ,

Normally if injection is not working it could be due to a few things:

  1. annotation not in place (not the case here)
  2. webhook certs not being correct (doesn’t appear to be a problem - we’d see it in the proxy logs in the injector if this was the case)
  3. something else blocking communication between injector and kubeapi.

To explore #3, can you dump the logs from the linkerd-injector container (not the linkerd-proxy container) on the linkerd-injector workload in the linkerd ns when you get a chance?

3

Sure can. I believe the logs above were from the proxy-injector container but will paste again.

Proxy Injector Container:

time="2024-07-03T15:01:33Z" level=info msg="running version edge-24.6.4"
time="2024-07-03T15:01:33Z" level=info msg="starting admin server on :9995"
time="2024-07-03T15:01:33Z" level=info msg="waiting for caches to sync"
time="2024-07-03T15:01:33Z" level=info msg="listening at :8443"
time="2024-07-03T15:01:33Z" level=info msg="caches synced"
4

Definitely odd. I assume you’ve cycled the workloads after adding the annotation at the pod’s spec.template.annotations level?

If so, then this suggests that there’s something going on between the kubeapi and the linkerd control plane where the injector is not getting the webhook calls for whatever reason. Do you have any network policies in place that may be in play?

5

Correct, I did do a restart of the deployment to see if it would trigger linkerd, but no luck.

I did wait while we finished setting up BGP, assuming maybe that was the culprit, but still having the same issues. Nothing outstanding on the networking side besides using HTTP/S Proxy and generic Calico v3.28.0 networking config with BGP configured. Using Ubuntu 22.04 and Kubernetes v1.30.2.

6

If you set the proxy injector logs to debug, does that provide any additional information? The proxy injector is a mutating admission webhook, so it’s called by Kubernetes. Either it is failing (which hopefully the debug logs will show) or it isn’t being called.

If the debug logs don’t show anything, there are some other tips for debugging webhooks like this in Kubernetes, e.g. setting the failure policy to fail. See How do I debug a Kubernetes validating admission webhook? - Stack Overflow for more.