# Stable-2.12.4 - Multicluster Headless Connection "504 Gateway Timeout"

**URL:** <https://linkerd.buoyant.io/t/stable-2-12-4-multicluster-headless-connection-504-gateway-timeout/114>\
**Category:** Linkerd General Discussion\
**Created:** [May 3, 2023, 2:47pm UTC](https://linkerd.buoyant.io/t/stable-2-12-4-multicluster-headless-connection-504-gateway-timeout/114 "2023-05-03T14:47:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![seb](https://avatars.discourse-cdn.com/v4/letter/s/8edcca/32.png) [@seb](https://linkerd.buoyant.io/u/seb)\
**Post date:** [May 3, 2023, 2:47pm UTC](https://linkerd.buoyant.io/t/stable-2-12-4-multicluster-headless-connection-504-gateway-timeout/114/1 "2023-05-03T14:47:58Z")

</div>

Hey,  
I try to setup a multicluster with headless service support using linkerd stable-2.12.4, which should allow me address statefulset pods directly from the remote cluster.

Followed the docs [here](https://linkerd.io/2.13/tasks/multicluster-using-statefulsets)I’m pretty sure the cluster connection worked before I linked the cluster with the headless option:

linkerd multicluster --cluster-name eu2 --set “enableHeadlessServices=true”

The linkerd + multicluster checks are all happy ( ✅) and I also see new endpoints being created in the remote cluster, when I scale up/down the statefulset in the remote cluster.But when I query the service or a specific pod, I get this error, even if DNS gets resolved to the local Endpoint IP:

```auto
< HTTP/1.1 504 Gateway Timeout
< l5d-proxy-error: Gateway service in fail-fast

```

I labeled this service to be exported with a no `ClusterIP`:

```auto
apiVersion: v1
kind: Service
metadata:
  labels:
    mirror.linkerd.io/exported: "true"
  name: linkedapp-svc
  namespace: multiregion
spec:
  clusterIP: None
  clusterIPs:
  - None
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 8765
  selector:
    app: linkedapp

```

Here is an example of the endpoints that can be resolved but throw the 504 on connection:

```auto
$ k get endpoints -n multiregion linkedapp-svc-eu2 -o yaml |k neatapiVersion: v1
kind: Endpoints
metadata:
  annotations:
    mirror.linkerd.io/remote-gateway-identity: linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
    mirror.linkerd.io/remote-svc-fq-name: linkedapp-svc.multiregion.svc.cluster.local
  labels:
    mirror.linkerd.io/cluster-name: eu2
    mirror.linkerd.io/mirrored-service: "true"
  name: linkedapp-svc-eu2
  namespace: multiregion
subsets:
- addresses:
  - hostname: linkedapp-1
    ip: 10.100.125.234
  - hostname: linkedapp-0
    ip: 10.100.220.47
  - hostname: linkedapp-2
    ip: 10.100.29.158
  ports:
  - port: 8765
    protocol: TCP

```

All this is running between two AWS EKS clusters running with `v1.22.17` with the default AWS VPC CNI.  
The classic NLB that Linkerd creates are running and accessible, I also let the recreate multiple times.  
I tried to do a clean reinstall, recreate the NLB but it doesn’t change ☹

I’ve found [this discussion](https://github.com/linkerd/linkerd2/issues/8934#issuecomment-1198378840), but as this is a fresh installation without any but my test traffic, that shouldn’t be the problem, no?

Any hints?  
Maybe a AWS CNI Issue?

---

<div class="post-metadata">

**Author:** ![Alex](https://yyz1.discourse-cdn.com/flex029/user_avatar/linkerd.buoyant.io/alex/32/63_2.png) [@Alex](https://linkerd.buoyant.io/u/Alex)\
**Post date:** [May 3, 2023, 6:59pm UTC](https://linkerd.buoyant.io/t/stable-2-12-4-multicluster-headless-connection-504-gateway-timeout/114/2 "2023-05-03T18:59:27Z")

</div>

Hi @seb

Are the requests that you’re making from the source cluster being made from inside a meshed pod? Because all multicluster traffic needs to be encrypted with mTLS, only meshed pods will be able to query the mirror services.

I’d also recommend taking a look at the linkerd-gateway logs in the target cluster. These logs may give you some clue as to why the gateway service is in fail-fast.

---

<div class="post-metadata">

**Author:** ![seb](https://avatars.discourse-cdn.com/v4/letter/s/8edcca/32.png) [@seb](https://linkerd.buoyant.io/u/seb)\
**Post date:** [May 4, 2023, 6:11am UTC](https://linkerd.buoyant.io/t/stable-2-12-4-multicluster-headless-connection-504-gateway-timeout/114/3 "2023-05-04T06:11:19Z")

</div>

Hi @Alex ,  
thanks for your response!🙂  
Yes both clusters have an identical namespace (“multiregion” in that case) and inject Linkerd to all pods within that namespace.

I’ll rebuild everything from scratch again and also have a look on the gateway logs. Keep you posted !

---

<div class="post-metadata">

**Author:** ![seb](https://avatars.discourse-cdn.com/v4/letter/s/8edcca/32.png) [@seb](https://linkerd.buoyant.io/u/seb)\
**Post date:** [May 10, 2023, 2:59pm UTC](https://linkerd.buoyant.io/t/stable-2-12-4-multicluster-headless-connection-504-gateway-timeout/114/4 "2023-05-10T14:59:02Z")

</div>

Hey @Alex ,

Just spend some more time on this topic, wiped again the complete installation and reinstalled it. But I got the same Issue.

Checking the linkerd-gateway logs in the target cluster I get the same issues as described here:

> **[Multi-Cluster communication connections must be mutually authenticated ·...](https://github.com/linkerd/linkerd2/discussions/8148)**
>
> Following steps I have performed: I have created 2 AKS clusters east and west. Create certs via //cert step certificate create root.linkerd.cluster.local root.crt root.key --profile root-ca --no-pa...

```auto
INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=192.168.100.175:50015

```

Any more ideas?  
I tried to fix the Health Probes on the AWS LB but it doesn’t help so far…

EDIT: Talking to the headless services is now working, finally 🥳  
In the end I had to restart all involved pods and now it’s working.

Anyhow, I still get the error above in the gateway. Even if they’re not blocking for Multicluster Communication, I’d still like to solve them. Is there a fix for the LB Health Probes, ready to be applied?

---

<div class="post-metadata">

**Author:** ![Alex](https://yyz1.discourse-cdn.com/flex029/user_avatar/linkerd.buoyant.io/alex/32/63_2.png) [@Alex](https://linkerd.buoyant.io/u/Alex)\
**Post date:** [May 10, 2023, 6:16pm UTC](https://linkerd.buoyant.io/t/stable-2-12-4-multicluster-headless-connection-504-gateway-timeout/114/5 "2023-05-10T18:16:00Z")

</div>

Glad to hear that you got multicluster traffic working!!!

As for the errors in the logs, we have an open issue tracking this ([Linkerd Gateway logs spammed with "connections must be mutually authenticated" from kube-system probes · Issue #10203 · linkerd/linkerd2 · GitHub](https://github.com/linkerd/linkerd2/issues/10203))