In 2.13, we introduced leader election for the policy-controller. The policy-controller now updates resource statuses (HttpRoute resources may contain parentRefs and backendRefs, the status indicates whether the references are valid). Only one controller may write/patch resources. Leader election is implemented using Lease objects.
It seems that Argo excludes the resource. Looking at their documentation, it seems that some groups are excluded by default (e.g events.k8s.io). I would perhaps double check that’s not the case for the coordination group.
In the documentation I linked, Argo also seems to support group inclusion, so you could try to explicitly include the group and see if on a subsequent deploy the policy-controller-write resource is created.
Thanks for clarifying the usage of the Lease object.
I found that all coordination.k8s.io objects are excluded by default by Argo, just like events.k8s.io which you already mentioned. I believe that you can’t change this default behavior (yet), by adding it to the inclusions for example.
We’ll see what we can do on our side. This seems to be a problem with Argo hardcoding exclusion groups though. As a workaround, you can manually create the lease resource.
We have a lot of k8s environments. That is one reason for using Argo. Manual steps do not really work. This was my final test before pushing Linkerd out to production environments. It has been working fine on test/staging environments so far. But had to make sure major upgrades of Linkerd also works.
We are on k8s 1.24.6 mostly and Argo 2.5.15. Hopefully we can find a way to make this deployment smooth also
I think this might be a behavioral difference or bug with the linkerd components. I run a few other apps that use kubernetes lease resources and they are deployed with Argocd without any manual steps. The other apps do not appear to have lease manifests applied to the cluster when installing them. Instead, it appears the apps themselves are creating the lease resource in the cluster and updating it. Since the app creates the lease object itself at run time, Argocd does not need to manage the object.
One of the apps that uses leases and does not have any issues being deployed with Argocd is cert-manager. Looking at the code, it appears cert-manager both creates the lease object if it does not exist and manages it automatically. The helm chart for cert-manager has an rbac role that allows the service account to create, get, update, and patch lease resources. Looking at the code for the component that uses leases, it calls leaderelection.NewLeaderElector(). This appears to create the lease object and refresh it. Cert-manager is using a common kubernetes go module, k8s.io/client-go/tools/leaderelection, that helps manage leases. The common go module also has a good example on how to use it.
@derektamsen I was just about to post the same thoughts The lease objects are really runtime objects and should not be part of the Helm chart. Sounds doable for Linkerd to fix this issue
There was an issue for this indeed, but that is in the Argo repository
I created an issue in the Linkerd repo, because I think that this issue can be fixed by Linkerd instead of by Argo.
@Linky Can you create a Linkerd issue with the information from your initial support post? The rest of us on this thread can then add our 2 cents to that issue. This is imho a Linkerd issue and not a ArgoCD issue
Thanks for bringing this to our attention. We’re currently reviewing options for improving interoperability between ArgoCD and 2.13’s introduction of the Lease resource. We’ll have an update on the plan very soon.
