# \[Cert-Manager\] Webhook Certificates renewal Failure

**URL:** https://linkerd.buoyant.io/t/cert-manager-webhook-certificates-renewal-failure/223
**Category:** Linkerd General Discussion
**Tags:** certificates
**Created:** [August 10, 2023, 10:20pm UTC](https://linkerd.buoyant.io/t/cert-manager-webhook-certificates-renewal-failure/223 "2023-08-10T22:20:12Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![pt](https://avatars.discourse-cdn.com/v4/letter/p/ccd318/32.png) [@pt](https://linkerd.buoyant.io/u/pt)
#### Post date: [August 10, 2023, 10:20pm UTC](https://linkerd.buoyant.io/t/cert-manager-webhook-certificates-renewal-failure/223/1 "2023-08-10T22:20:12Z")

</div>

When performing a `linkerd check --proxy` we’re experiencing an error `cert is not issued by the trust anchor: x509: certificate signed by unknown authority` which doesn’t make much sense to me. To my understanding, webhook certificates should not be issued by the root cert.

```auto
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
× proxy-injector webhook has valid cert
    cert is not issued by the trust anchor: x509: certificate signed by unknown authority
    see https://linkerd.io/2.13/checks/#l5d-proxy-injector-webhook-cert-valid for hints

linkerd-viz
-----------
√ linkerd-viz Namespace exists
× can initialize the client
    no running pods found for metrics-api
    see https://linkerd.io/2.13/checks/#l5d-viz-existence-client for hints

Status check results are ×
❯ linkerd check --proxy
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
× proxy-injector webhook has valid cert
    cert is not issued by the trust anchor: x509: certificate signed by unknown authority
    see https://linkerd.io/2.13/checks/#l5d-proxy-injector-webhook-cert-valid for hints

linkerd-viz
-----------
√ linkerd-viz Namespace exists
× can initialize the client
    no running pods found for metrics-api
    see https://linkerd.io/2.13/checks/#l5d-viz-existence-client for hints

Status check results are ×

```

We’re currently using Helm to install Linkerd and cert-manager to manage the certificates.

Also, the certificates is no different from what is described [create-issuers-referencing-the-secrets](https://linkerd.io/2.13/tasks/automatically-rotating-webhook-tls-credentials/#create-issuers-referencing-the-secrets) **(we’re using clusterIssuer instead of issuer)**

The only one off is we’re using cert-manager create/manage the cert for `webhook.linkerd.cluster.local` instead of relying on the `step` cli to create the secret [here](https://linkerd.io/2.13/tasks/automatically-rotating-webhook-tls-credentials/#save-the-signing-key-pair-as-a-secret).

On our initial `helm install` all check statuses passed when performing `linkerd check --proxy`. It is only an issue when we delete the webhook’s secrets after performing a `cmctl renew linkerd-proxy-injector -n linkerd`

Now, if I perform a `helm upgrade` things are back to normal. I’m hoping someone can help provide some insight into this thanks!

---

<div class="post-metadata">

### Author: ![pt](https://avatars.discourse-cdn.com/v4/letter/p/ccd318/32.png) [@pt](https://linkerd.buoyant.io/u/pt)
#### Post date: [August 11, 2023, 11:10pm UTC](https://linkerd.buoyant.io/t/cert-manager-webhook-certificates-renewal-failure/223/2 "2023-08-11T23:10:19Z")

</div>

This GitHub [issue](https://github.com/linkerd/linkerd2/issues/7519) seems similar except that the error message I’m getting is different, also restarting the Linkerd pods doesn’t resolve this.

---

<div class="post-metadata">

### Author: ![alpeb](https://yyz1.discourse-cdn.com/flex029/user_avatar/linkerd.buoyant.io/alpeb/32/61_2.png) [@alpeb](https://linkerd.buoyant.io/u/alpeb)
#### Post date: [August 14, 2023, 2:29pm UTC](https://linkerd.buoyant.io/t/cert-manager-webhook-certificates-renewal-failure/223/3 "2023-08-14T14:29:21Z")

</div>

The injector webhook cert should be rooted at a CA bundle that is provided in the “linkerd-proxy-injector-webhook-config” MutatingWebhookConfiguration. This is configured via the `proxyInjector.caBundle` helm value. In the [docs](https://linkerd.io/2.13/tasks/automatically-rotating-webhook-tls-credentials/) you’ll see its payload is also used to generate the “webhook-issuer-tls” secret, that you should refer to in cert-manager’s Issuer resource.

---

<div class="post-metadata">

### Author: ![pt](https://avatars.discourse-cdn.com/v4/letter/p/ccd318/32.png) [@pt](https://linkerd.buoyant.io/u/pt)
#### Post date: [August 14, 2023, 6:37pm UTC](https://linkerd.buoyant.io/t/cert-manager-webhook-certificates-renewal-failure/223/4 "2023-08-14T18:37:13Z")

</div>

@alpeb thank you very much for your reply. The docs provided are what I’ve followed, although all pods are running it’s only when I run a `linkerd check` that the `proxy-injector webhook` error is presented.

After further investigation, I found this error (below) in the cert-manager logs. The fix was to add an annotation `cert-manager.io/allow-direct-injection: "true"` to the `secrets`

```auto

cert-manager: could not find any ca data in data source for target","v":2,"kind":"mutatingwebhookconfiguration"

```

---

<div class="post-metadata">

### Author: ![alpeb](https://yyz1.discourse-cdn.com/flex029/user_avatar/linkerd.buoyant.io/alpeb/32/61_2.png) [@alpeb](https://linkerd.buoyant.io/u/alpeb)
#### Post date: [August 16, 2023, 2:13pm UTC](https://linkerd.buoyant.io/t/cert-manager-webhook-certificates-renewal-failure/223/5 "2023-08-16T14:13:23Z")

</div>

Ah great find, thanks for coming back with the resolution, I’m sure that’ll be helpful for others as well 🙂
