[Cert-Manager] Webhook Certificates renewal Failure

Linkerd General Discussion
1

When performing a linkerd check --proxy we’re experiencing an error cert is not issued by the trust anchor: x509: certificate signed by unknown authority which doesn’t make much sense to me. To my understanding, webhook certificates should not be issued by the root cert.

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
× proxy-injector webhook has valid cert
    cert is not issued by the trust anchor: x509: certificate signed by unknown authority
    see https://linkerd.io/2.13/checks/#l5d-proxy-injector-webhook-cert-valid for hints

linkerd-viz
-----------
√ linkerd-viz Namespace exists
× can initialize the client
    no running pods found for metrics-api
    see https://linkerd.io/2.13/checks/#l5d-viz-existence-client for hints

Status check results are ×
❯ linkerd check --proxy
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
× proxy-injector webhook has valid cert
    cert is not issued by the trust anchor: x509: certificate signed by unknown authority
    see https://linkerd.io/2.13/checks/#l5d-proxy-injector-webhook-cert-valid for hints

linkerd-viz
-----------
√ linkerd-viz Namespace exists
× can initialize the client
    no running pods found for metrics-api
    see https://linkerd.io/2.13/checks/#l5d-viz-existence-client for hints

Status check results are ×

We’re currently using Helm to install Linkerd and cert-manager to manage the certificates.

Also, the certificates is no different from what is described create-issuers-referencing-the-secrets (we’re using clusterIssuer instead of issuer)

The only one off is we’re using cert-manager create/manage the cert for webhook.linkerd.cluster.local instead of relying on the step cli to create the secret here.

On our initial helm install all check statuses passed when performing linkerd check --proxy. It is only an issue when we delete the webhook’s secrets after performing a cmctl renew linkerd-proxy-injector -n linkerd

Now, if I perform a helm upgrade things are back to normal. I’m hoping someone can help provide some insight into this thanks!

2

This GitHub issue seems similar except that the error message I’m getting is different, also restarting the Linkerd pods doesn’t resolve this.

3

The injector webhook cert should be rooted at a CA bundle that is provided in the “linkerd-proxy-injector-webhook-config” MutatingWebhookConfiguration. This is configured via the proxyInjector.caBundle helm value. In the docs you’ll see its payload is also used to generate the “webhook-issuer-tls” secret, that you should refer to in cert-manager’s Issuer resource.

4

@alpeb thank you very much for your reply. The docs provided are what I’ve followed, although all pods are running it’s only when I run a linkerd check that the proxy-injector webhook error is presented.

After further investigation, I found this error (below) in the cert-manager logs. The fix was to add an annotation cert-manager.io/allow-direct-injection: "true" to the secrets


cert-manager: could not find any ca data in data source for target","v":2,"kind":"mutatingwebhookconfiguration"
5

Ah great find, thanks for coming back with the resolution, I’m sure that’ll be helpful for others as well :slight_smile: