Hi everyone,
I have followed Pod-to-Pod Multi-cluster after installing linkerd service mash for flat network, and deploying and labelling sample application(mirror.linkerd.io/exported=remote-discovery), I can’t see service getting mirrored in destination cluster.
Thanks for checking out the feature @anvit. Can you confirm what state the service mirror was in? And do you by any chance have service mirror logs we can take a look at?
Hey @Alen , thanks attaching service mirror pod logs and pod status.
[ 0.002168s] INFO ThreadId(01) linkerd2_proxy: release 2.238.0 (99626eb) by linkerd on 2024-06-26T03:48:04Z
[ 0.004883s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[ 0.005966s] INFO ThreadId(01) linkerd2_proxy: Admin interface on [::]:4191
[ 0.005977s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on [::]:4143
[ 0.005980s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[ 0.005983s] INFO ThreadId(01) linkerd2_proxy: Tap interface on [::]:4190
[ 0.005986s] INFO ThreadId(01) linkerd2_proxy: SNI is linkerd-service-mirror-target-one.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
[ 0.005989s] INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-service-mirror-target-one.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
[ 0.005992s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.007461s] INFO ThreadId(01) dst:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_pool_p2c: Adding endpoint addr=10.0.155.125:8086
[ 0.007681s] INFO ThreadId(01) policy:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_pool_p2c: Adding endpoint addr=10.0.155.125:8090
[ 0.007839s] INFO ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_pool_p2c: Adding endpoint addr=10.0.153.26:8080
[ 0.025817s] INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=linkerd-service-mirror-target-one.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
[ 0.204508s] INFO ThreadId(01) outbound:proxy{addr=172.20.0.1:443}:balance{addr=kubernetes.default.svc.cluster.local:443}: linkerd_pool_p2c: Adding endpoint addr=10.0.133.215:443
[ 0.204547s] INFO ThreadId(01) outbound:proxy{addr=172.20.0.1:443}:balance{addr=kubernetes.default.svc.cluster.local:443}: linkerd_pool_p2c: Adding endpoint addr=10.0.147.80:443
[ 1134.901800s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=10.0.142.41:50604 server.addr=192.168.57.233:443
[ 1136.904344s] INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=10.0.142.41:50618 server.addr=192.168.57.233:443
it does look like discovery is not working for some reason. We see local endpoints being tracked and added (for the linkerd control plane components) but nothing from the remote end.
Can you confirm that your service mirror can reach the remote cluster’s kubeapi from a layer3 routing perspective? In order to mirror services the sm on east needs to be able to set up a watch with the k8s api on west, which requires flat networking/l3 routability between the two, and no network policies that block the traffic
@Alen ,yes it was unable to reach to kube-api server, updated networking configuration, it worked for us.
Thanks.
