# Linkerd CNI in AKS fails after Calico pod get restarted

**URL:** <https://linkerd.buoyant.io/t/linkerd-cni-in-aks-fails-after-calico-pod-get-restarted/298>\
**Category:** Linkerd General Discussion\
**Created:** [October 23, 2023, 2:50pm UTC](https://linkerd.buoyant.io/t/linkerd-cni-in-aks-fails-after-calico-pod-get-restarted/298 "2023-10-23T14:50:59Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mohammed.elmaleeh](https://avatars.discourse-cdn.com/v4/letter/m/59ef9b/32.png) [@mohammed.elmaleeh](https://linkerd.buoyant.io/u/mohammed.elmaleeh)\
**Post date:** [October 23, 2023, 2:50pm UTC](https://linkerd.buoyant.io/t/linkerd-cni-in-aks-fails-after-calico-pod-get-restarted/298/1 "2023-10-23T14:50:59Z")

</div>

Hello All,

We use Linkerd `stable-2.14.1` with Linkerd CNI in AKS(1.25.6) with Calico CNI. We started getting issues when Calico pod in one AKS node get restarted, what happen next is the Linkerd CNI won’t be available on that node that means no iptables rules which also means pods will not be able to proxy the connection through the Linkerd side car proxies. The fix we do is to restart LInkerd CNI which will update the CNI configuration in the node to include it again.  
I was expecting that the CNI watches `/host/etc/cni/net.d/10-calico.conflist` where if it changed by other CNI (in this case Calico) it re-apply Linkerd CNI again, but this doesn’t happen and we have to do that manually, what can we do to over come this issue.

Linkerd CNI pod logs:

```auto
Wrote linkerd CNI binaries to /host/opt/cni/bin
Installing CNI configuration in "chained" mode for /host/etc/cni/net.d/10-calico.conflist
Using CNI config template from CNI_NETWORK_CONFIG environment variable.
      "k8s_api_root": "https:// __KUBERNETES_SERVICE_HOST__ : __KUBERNETES_SERVICE_PORT__",
      "k8s_api_root": "https://X.X.X.X: __KUBERNETES_SERVICE_PORT__",
CNI config: {
  "name": "linkerd-cni",
  "type": "linkerd-cni",
  "log_level": "debug",
  "policy": {
      "type": "k8s",
      "k8s_api_root": "https://X.X.X.X:443",
      "k8s_auth_token": " __SERVICEACCOUNT_TOKEN__"
  },
  "kubernetes": {
      "kubeconfig": "/etc/cni/net.d/ZZZ-linkerd-cni-kubeconfig"
  },
  "linkerd": {
    "incoming-proxy-port": 4143,
    "outgoing-proxy-port": 4140,
    "proxy-uid": 2102,
    "ports-to-redirect": [],
    "inbound-ports-to-ignore": ["4191","4190"],
    "simulate": false,
    "use-wait-flag": true
  }
}
Created CNI config /host/etc/cni/net.d/10-calico.conflist
Setting up watches.
Watches established.

```

Linkerd CNI pod after a successful installation and operating as expected:

```auto
root@linkerd-cni-f9cgt:/linkerd# cat /host/etc/cni/net.d/10-calico.conflist | jq .plugins[].type
"calico"
"bandwidth"
"portmap"
"linkerd-cni"

```

Linkerd CNI pod after Calico pod in that node get restarted:

```auto
root@linkerd-cni-4adat:/linkerd# cat /host/etc/cni/net.d/10-calico.conflist | jq .plugins[].type
"calico"
"bandwidth"
"portmap"

```

---

<div class="post-metadata">

**Author:** ![mohammed.elmaleeh](https://avatars.discourse-cdn.com/v4/letter/m/59ef9b/32.png) [@mohammed.elmaleeh](https://linkerd.buoyant.io/u/mohammed.elmaleeh)\
**Post date:** [October 23, 2023, 9:19pm UTC](https://linkerd.buoyant.io/t/linkerd-cni-in-aks-fails-after-calico-pod-get-restarted/298/2 "2023-10-23T21:19:13Z")

</div>

After more investigation I found out that the Linkerd CNI script monitor only a specific events (CREATE & DELETE) that doesn’t get triggered when calico pod restarted.

```auto
monitor() {
  inotifywait -m "${HOST_CNI_NET}" -e create,delete |
    while read -r directory action filename; do
      if [["$filename" =~ .*.(conflist|conf)$]]; then 
        echo "Detected change in $directory: $action $filename"
        sync "$filename" "$action" "$cni_conf_sha"
        # When file exists (i.e we didn't deal with a DELETE ev)
        # then calculate its sha to be used the next turn.
        if [[-e "$directory/$filename" && "$action" != 'DELETE']]; then
          cni_conf_sha="$(sha256sum "$directory/$filename" | while read -r s _; do echo "$s"; done)"
        fi
      fi
    done
}

```

These are the events on the CNI directory when I try reproducing the issue by killing calico-node in the node where Linkerd CNI run

```auto
root@linkerd-cni-9h2rc:/linkerd $ inotifywait -m /host/etc/cni/net.d                        
Setting up watches.
Watches established.
/host/etc/cni/net.d/ OPEN,ISDIR 
/host/etc/cni/net.d/ CLOSE_NOWRITE,CLOSE,ISDIR 
/host/etc/cni/net.d/ MODIFY calico-kubeconfig
/host/etc/cni/net.d/ OPEN calico-kubeconfig
/host/etc/cni/net.d/ MODIFY calico-kubeconfig
/host/etc/cni/net.d/ CLOSE_WRITE,CLOSE calico-kubeconfig
/host/etc/cni/net.d/ OPEN,ISDIR 
/host/etc/cni/net.d/ ACCESS,ISDIR 
/host/etc/cni/net.d/ MODIFY 10-calico.conflist
/host/etc/cni/net.d/ OPEN 10-calico.conflist
/host/etc/cni/net.d/ ACCESS,ISDIR 
/host/etc/cni/net.d/ CLOSE_NOWRITE,CLOSE,ISDIR 
/host/etc/cni/net.d/ MODIFY 10-calico.conflist
/host/etc/cni/net.d/ CLOSE_WRITE,CLOSE 10-calico.conflist
/host/etc/cni/net.d/ OPEN 10-calico.conflist
/host/etc/cni/net.d/ ACCESS 10-calico.conflist
/host/etc/cni/net.d/ CLOSE_NOWRITE,CLOSE 10-calico.conflist
/host/etc/cni/net.d/ OPEN 10-calico.conflist
/host/etc/cni/net.d/ ACCESS 10-calico.conflist
/host/etc/cni/net.d/ CLOSE_NOWRITE,CLOSE 10-calico.conflist
/host/etc/cni/net.d/ OPEN,ISDIR 
/host/etc/cni/net.d/ ACCESS,ISDIR 
/host/etc/cni/net.d/ CLOSE_NOWRITE,CLOSE,ISDIR 
/host/etc/cni/net.d/ OPEN 10-calico.conflist
/host/etc/cni/net.d/ ACCESS 10-calico.conflist
/host/etc/cni/net.d/ CLOSE_NOWRITE,CLOSE 10-calico.conflist
/host/etc/cni/net.d/ OPEN,ISDIR 
/host/etc/cni/net.d/ ACCESS,ISDIR 
/host/etc/cni/net.d/ CLOSE_NOWRITE,CLOSE,ISDIR 
/host/etc/cni/net.d/ OPEN 10-calico.conflist
/host/etc/cni/net.d/ ACCESS 10-calico.conflist
/host/etc/cni/net.d/ CLOSE_NOWRITE,CLOSE 10-calico.conflist
/host/etc/cni/net.d/ OPEN,ISDIR 
/host/etc/cni/net.d/ CLOSE_NOWRITE,CLOSE,ISDIR 
/host/etc/cni/net.d/ MODIFY calico-kubeconfig
/host/etc/cni/net.d/ OPEN calico-kubeconfig
/host/etc/cni/net.d/ MODIFY calico-kubeconfig
/host/etc/cni/net.d/ CLOSE_WRITE,CLOSE calico-kubeconfig
/host/etc/cni/net.d/ OPEN,ISDIR 
/host/etc/cni/net.d/ ACCESS,ISDIR 
/host/etc/cni/net.d/ CLOSE_NOWRITE,CLOSE,ISDIR 
/host/etc/cni/net.d/ OPEN 10-calico.conflist
/host/etc/cni/net.d/ ACCESS 10-calico.conflist
/host/etc/cni/net.d/ CLOSE_NOWRITE,CLOSE 10-calico.conflist
/host/etc/cni/net.d/ OPEN,ISDIR 
/host/etc/cni/net.d/ ACCESS,ISDIR 
/host/etc/cni/net.d/ ACCESS,ISDIR 
/host/etc/cni/net.d/ CLOSE_NOWRITE,CLOSE,ISDIR 
/host/etc/cni/net.d/ OPEN 10-calico.conflist
/host/etc/cni/net.d/ ACCESS 10-calico.conflist
/host/etc/cni/net.d/ CLOSE_NOWRITE,CLOSE 10-calico.conflist

```

I guess if the Linkerd CNI script monitor also `modify` events that could solve this issue.

---

<div class="post-metadata">

**Author:** ![matei](https://avatars.discourse-cdn.com/v4/letter/m/e495f1/32.png) [@matei](https://linkerd.buoyant.io/u/matei)\
**Post date:** [October 30, 2023, 4:01pm UTC](https://linkerd.buoyant.io/t/linkerd-cni-in-aks-fails-after-calico-pod-get-restarted/298/3 "2023-10-30T16:01:25Z")

</div>

Hi @mohammed.elmaleeh, thanks for the write-up! I had a look through [calico’s code](https://github.com/projectcalico/calico/blame/b0a05c27f6df21e8b84147aca96b4ee106f0bb2a/cni-plugin/pkg/install/install.go#L394) and indeed it does seem that they do a `write()`. Most distributions do a write to a temp file and a move (when modifying the configuration file); we already cover these two events [in the CNI plugin code](https://github.com/linkerd/linkerd2-proxy-init/blob/main/cni-plugin/deployment/scripts/install-cni.sh#L266-L283) (_Note_: if you looked at the code in the linkerd2 repo, we’ve actually moved the CNI installer to a different location).

It would make sense to me to add `modify`. I’m surprised people haven’t run into this before. Do you mind creating an issue in the linkerd2 repo?

---

<div class="post-metadata">

**Author:** ![mohammed.elmaleeh](https://avatars.discourse-cdn.com/v4/letter/m/59ef9b/32.png) [@mohammed.elmaleeh](https://linkerd.buoyant.io/u/mohammed.elmaleeh)\
**Post date:** [October 31, 2023, 10:12am UTC](https://linkerd.buoyant.io/t/linkerd-cni-in-aks-fails-after-calico-pod-get-restarted/298/4 "2023-10-31T10:12:27Z")

</div>

Hi @matei, thanks for your reply and the thorough investigation. Here you go

> <https://github.com/linkerd/linkerd2/issues/11549>
>
> \### What is the issue?
> 
> We started getting issues when Calico pod in one AKS nod…e get restarted, what happen next is that Calico will rewrite the CNI (/host/etc/cni/net.d/10-calico.conflist) and remove Linkerd CNI plugin configuration, and the monitor script fail to detect this change as it catches only \`CREATE\` and \`DELETE\` events while Calico do \`modify\` the file. 
> 
> \### How can it be reproduced?
> 
> Restarting Calico pod in an AKS node that has Linkerd installed with Linkerd CNI plugin in chaining mode.
> 
> 
> \### Logs, error output, etc
> 
> Events that happens on cni directory when Calico pod get restarted:
> \`\`\`
> root@linkerd-cni-9h2rc:/linkerd $ inotifywait -m /host/etc/cni/net.d                        
> Setting up watches.
> Watches established.
> /host/etc/cni/net.d/ OPEN,ISDIR 
> /host/etc/cni/net.d/ CLOSE\_NOWRITE,CLOSE,ISDIR 
> /host/etc/cni/net.d/ MODIFY calico-kubeconfig
> /host/etc/cni/net.d/ OPEN calico-kubeconfig
> /host/etc/cni/net.d/ MODIFY calico-kubeconfig
> /host/etc/cni/net.d/ CLOSE\_WRITE,CLOSE calico-kubeconfig
> /host/etc/cni/net.d/ OPEN,ISDIR 
> /host/etc/cni/net.d/ ACCESS,ISDIR 
> /host/etc/cni/net.d/ MODIFY 10-calico.conflist
> /host/etc/cni/net.d/ OPEN 10-calico.conflist
> /host/etc/cni/net.d/ ACCESS,ISDIR 
> /host/etc/cni/net.d/ CLOSE\_NOWRITE,CLOSE,ISDIR 
> /host/etc/cni/net.d/ MODIFY 10-calico.conflist
> /host/etc/cni/net.d/ CLOSE\_WRITE,CLOSE 10-calico.conflist
> /host/etc/cni/net.d/ OPEN 10-calico.conflist
> /host/etc/cni/net.d/ ACCESS 10-calico.conflist
> /host/etc/cni/net.d/ CLOSE\_NOWRITE,CLOSE 10-calico.conflist
> /host/etc/cni/net.d/ OPEN 10-calico.conflist
> /host/etc/cni/net.d/ ACCESS 10-calico.conflist
> /host/etc/cni/net.d/ CLOSE\_NOWRITE,CLOSE 10-calico.conflist
> /host/etc/cni/net.d/ OPEN,ISDIR 
> /host/etc/cni/net.d/ ACCESS,ISDIR 
> /host/etc/cni/net.d/ CLOSE\_NOWRITE,CLOSE,ISDIR 
> /host/etc/cni/net.d/ OPEN 10-calico.conflist
> /host/etc/cni/net.d/ ACCESS 10-calico.conflist
> /host/etc/cni/net.d/ CLOSE\_NOWRITE,CLOSE 10-calico.conflist
> /host/etc/cni/net.d/ OPEN,ISDIR 
> /host/etc/cni/net.d/ ACCESS,ISDIR 
> /host/etc/cni/net.d/ CLOSE\_NOWRITE,CLOSE,ISDIR 
> /host/etc/cni/net.d/ OPEN 10-calico.conflist
> /host/etc/cni/net.d/ ACCESS 10-calico.conflist
> /host/etc/cni/net.d/ CLOSE\_NOWRITE,CLOSE 10-calico.conflist
> /host/etc/cni/net.d/ OPEN,ISDIR 
> /host/etc/cni/net.d/ CLOSE\_NOWRITE,CLOSE,ISDIR 
> /host/etc/cni/net.d/ MODIFY calico-kubeconfig
> /host/etc/cni/net.d/ OPEN calico-kubeconfig
> /host/etc/cni/net.d/ MODIFY calico-kubeconfig
> /host/etc/cni/net.d/ CLOSE\_WRITE,CLOSE calico-kubeconfig
> /host/etc/cni/net.d/ OPEN,ISDIR 
> /host/etc/cni/net.d/ ACCESS,ISDIR 
> /host/etc/cni/net.d/ CLOSE\_NOWRITE,CLOSE,ISDIR 
> /host/etc/cni/net.d/ OPEN 10-calico.conflist
> /host/etc/cni/net.d/ ACCESS 10-calico.conflist
> /host/etc/cni/net.d/ CLOSE\_NOWRITE,CLOSE 10-calico.conflist
> /host/etc/cni/net.d/ OPEN,ISDIR 
> /host/etc/cni/net.d/ ACCESS,ISDIR 
> /host/etc/cni/net.d/ ACCESS,ISDIR 
> /host/etc/cni/net.d/ CLOSE\_NOWRITE,CLOSE,ISDIR 
> /host/etc/cni/net.d/ OPEN 10-calico.conflist
> /host/etc/cni/net.d/ ACCESS 10-calico.conflist
> /host/etc/cni/net.d/ CLOSE\_NOWRITE,CLOSE 10-calico.conflist
> \`\`\`
> 
> \### output of \`linkerd check -o short\`
> 
> \`\`\`
> linkerd-identity
> \----------------
> ‼ trust anchors are valid for at least 60 days
> Anchors expiring soon:
> \* 6819992339580814618406831927340696977 root.mpdd3weuaksplatfogene001.linkerd.dev.mpd.corp will expire on 2023-12-18T11:44:36Z
> see https://linkerd.io/2.14/checks/#l5d-identity-trustAnchors-not-expiring-soon for hints
> ‼ issuer cert is valid for at least 60 days
> issuer certificate will expire on 2023-11-01T12:14:53Z
> see https://linkerd.io/2.14/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
> 
> linkerd-webhooks-and-apisvc-tls
> \-------------------------------
> ‼ proxy-injector cert is valid for at least 60 days
> certificate will expire on 2023-11-01T04:00:46Z
> see https://linkerd.io/2.14/checks/#l5d-proxy-injector-webhook-cert-not-expiring-soon for hints
> ‼ sp-validator cert is valid for at least 60 days
> certificate will expire on 2023-11-01T04:00:45Z
> see https://linkerd.io/2.14/checks/#l5d-sp-validator-webhook-cert-not-expiring-soon for hints
> ‼ policy-validator cert is valid for at least 60 days
> certificate will expire on 2023-11-01T04:00:46Z
> see https://linkerd.io/2.14/checks/#l5d-policy-validator-webhook-cert-not-expiring-soon for hints
> 
> control-plane-version
> \---------------------
> ‼ control plane is up-to-date
> is running version 2.14.1 but the latest stable version is 2.14.2
> see https://linkerd.io/2.14/checks/#l5d-version-control for hints
> ‼ control plane and cli versions match
> control plane running stable-2.14.1 but cli running stable-2.14.2
> see https://linkerd.io/2.14/checks/#l5d-version-control for hints
> 
> linkerd-control-plane-proxy
> \---------------------------
> ‼ control plane proxies are up-to-date
> some proxies are not running the current version:
> \* linkerd-destination-584bdf6db5-46bqr (stable-2.14.1)
> \* linkerd-destination-584bdf6db5-8vkbd (stable-2.14.1)
> \* linkerd-destination-584bdf6db5-z6rp5 (stable-2.14.1)
> \* linkerd-identity-bf6d68dbd-6fl2t (stable-2.14.1)
> \* linkerd-identity-bf6d68dbd-m7crm (stable-2.14.1)
> \* linkerd-identity-bf6d68dbd-mp5kn (stable-2.14.1)
> \* linkerd-proxy-injector-5dc9845984-48nzj (stable-2.14.1)
> \* linkerd-proxy-injector-5dc9845984-rwd8x (stable-2.14.1)
> \* linkerd-proxy-injector-5dc9845984-w8246 (stable-2.14.1)
> see https://linkerd.io/2.14/checks/#l5d-cp-proxy-version for hints
> ‼ control plane proxies and cli versions match
> linkerd-destination-584bdf6db5-46bqr running stable-2.14.1 but cli running stable-2.14.2
> see https://linkerd.io/2.14/checks/#l5d-cp-proxy-cli-version for hints
> 
> linkerd-ha-checks
> \-----------------
> ‼ pod injection disabled on kube-system
> kube-system namespace needs to have the label config.linkerd.io/admission-webhooks: disabled if injector webhook failure policy is Fail
> see https://linkerd.io/2.14/checks/#l5d-injection-disabled for hints
> 
> linkerd-viz
> \-----------
> ‼ tap API server cert is valid for at least 60 days
> certificate will expire on 2023-11-01T04:45:24Z
> see https://linkerd.io/2.14/checks/#l5d-tap-cert-not-expiring-soon for hints
> ‼ viz extension proxies are up-to-date
> some proxies are not running the current version:
> \* metrics-api-5b6f8959d4-wrq26 (stable-2.14.1)
> \* prometheus-9f6b88f65-npx4m (stable-2.14.1)
> \* tap-7fc76564c7-dskv6 (stable-2.14.1)
> \* tap-7fc76564c7-sxmt9 (stable-2.14.1)
> \* tap-7fc76564c7-txwlz (stable-2.14.1)
> \* tap-injector-77976df687-tzktl (stable-2.14.1)
> \* web-56c74b9989-97959 (stable-2.14.1)
> see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cp-version for hints
> ‼ viz extension proxies and cli versions match
> metrics-api-5b6f8959d4-wrq26 running stable-2.14.1 but cli running stable-2.14.2
> see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cli-version for hints
> 
> Status check results are √
> \`\`\`
> 
> \### Environment
> 
> \- Kubernetes Version: 1.25.6
> \- Cluster Environment: AKS
> \- Host OS: Linux
> \- Linkerd version: stable-2.14.1 
> 
> \### Possible solution
> 
> Adjust the \[monitor()\](https://github.com/linkerd/linkerd2-proxy-init/blob/main/cni-plugin/deployment/scripts/install-cni.sh#L287) to watch out for \`modify\` events.
> 
> \### Additional context
> 
> \_No response\_
> 
> \### Would you like to work on fixing this bug?
> 
> maybe