Hi all, is it not possible to only allow meshed connection to opencensus receiver for linkerd distributed tracing?
I have tried to only allow meshed connection but when any linkerd proxy is restarted, there seems to be a lot of unmeshed connection (from non-pod IP) to the opencensus receiver -probably due to retry since the connection is refused- until the connection is allowed.
here is the metric of tcp write when it happens
hi @rucciva
If I’m understanding you correctly, you’re seeing connections from a non-pod IP to the opencensus receiver and those connections are being refused. What is the source of those connections? If the source is not a pod… what is it?
And what is the behavior that you want here? If you want to only allow meshed connections, then isn’t rejecting connections from non-meshed sources the desired behavior?
If I’m understanding you correctly, you’re seeing connections from a non-pod IP to the opencensus receiver and those connections are being refused. What is the source of those connections? If the source is not a pod… what is it?
I’m not sure either, but it does falls on the IP range given to kubernetes. Maybe from kubernetes api server? What i’m sure is that it happens only when an injected pod get created/restarted. In my cluster, only linkerd proxy and linkerd control plane are configured to use the opencensus receiver.
And what is the behavior that you want here? If you want to only allow meshed connections, then isn’t rejecting connections from non-meshed sources the desired behavior?
Yes, i want to only allow meshed connection to opencensus receiver, but what i don’t want is to have a flood of denied tcp connection to opencensus server due to possible miss-configuration, because it consume a lot of bandwith and produce a lot of logs. Since the opencensus server is only used by linkerd, i suspect that it is related to it.
And, what i noticed is that from the github repo, it seems that the default policy is to allow any connection to opencensus receiver (https://github.com/linkerd/linkerd2/blob/main/jaeger/charts/linkerd-jaeger/templates/tracing-policy.yaml#L179-L180).
Ah, I see. I think there are two things which I would recommend investigating:
linkerd diagnostics proxy-metrics command. In particular, you’re looking for the inbound request/response count metrics and looking at the labels. The labels will illustrate which Server and Authorization the request matched.Hi, after running tcpdump from the node, i saw that packets are coming from the ip of the node and gets translated to ip that falls on the pod cidr, which act like a gateway for the pod cidr. But i can’t any host process that connect to the port of opencensus receiver using netstat or ss.
But now i’m thinking, do you know what would be the source ip of packet from linkerd-proxy that is injected to the opencensus collector itself when it needs to send telemetry data to the cluster ip of service pointed to the opencensus collector? Since that will become a u-turn packet, is it possible that the CNI impelementation rewrite the source ip as the node ip?
Running tcpdump directly from the node may give confusing or misleading results because pods run in different network namespaces. This talk from Kubecon talks about using nsenter to run commands like tcpdump in the pod’s network namespace after SSHing to a node and might be helpful:
i did run tcpdump in the pod using kubectl debug and see the same connection from host ip, but as before, no processes are using the outgoing port recorded in tcpdump.