Hi folks, I have a question regarding the tcp_{read/write}_bytes_total metrics, specifically about the different peer labels and the values that these metrics are returning.
To provide some context, these metrics are coming from a fresh Nginx pod with only the Linkerd sidecar, so there are no other network transfers that would be influencing the metric value. An Istio gateway exposes this pod and only one curl was made to the public endpoint. Before the curl request was made there were no metrics reported and following the request, there were no more increases in the value reported by these metrics.
To illustrate these are the metric results in question:
---- Read TCP metrics ----
linkerd_tcp_read_bytes_total{app="nginx", client_id="istio-ingressgateway-service-account.istio-system.serviceaccount.identity.linkerd.cluster.local", direction="inbound", instance="10.116.0.82:4191", job="linkerd-metrics", kubernetes_namespace="ns-rgnln8qp5g4g", kubernetes_pod_name="nginx-db4c665b9-4hhhq", peer="src", server_id="unknown", srv_kind="default", srv_name="all-unauthenticated", target_addr="10.116.0.82:80", target_ip="10.116.0.82", target_port="80", tls="true"} 436
linkerd_tcp_read_bytes_total{app="nginx", client_id="unknown", direction="inbound", instance="10.116.0.82:4191", job="linkerd-metrics", kubernetes_namespace="ns-rgnln8qp5g4g", kubernetes_pod_name="nginx-db4c665b9-4hhhq", no_tls_reason="loopback", peer="dst", server_id="unknown", tls="no_identity"} 853
---- Write TCP metrics ----
linkerd_tcp_write_bytes_total{app="nginx", client_id="istio-ingressgateway-service-account.istio-system.serviceaccount.identity.linkerd.cluster.local", direction="inbound", instance="10.116.0.82:4191", job="linkerd-metrics", kubernetes_namespace="ns-rgnln8qp5g4g", kubernetes_pod_name="nginx-db4c665b9-4hhhq", peer="src", server_id="unknown", srv_kind="default", srv_name="all-unauthenticated", target_addr="10.116.0.82:80", target_ip="10.116.0.82", target_port="80", tls="true"} 797
linkerd_tcp_write_bytes_total{app="nginx", client_id="unknown", direction="inbound", instance="10.116.0.82:4191", job="linkerd-metrics", kubernetes_namespace="ns-rgnln8qp5g4g", kubernetes_pod_name="nginx-db4c665b9-4hhhq", no_tls_reason="loopback", peer="dst", server_id="unknown", tls="no_identity"} 558
From my understanding tcp_read and peer=src should be the amount of data flowing from the pod to the client (ie HTTP response) and peer=dst is the data flowing from the client to the pod (ie HTTP request). With tcp_write and peer=src this should be equivalent to tcp_read and peer=dst, and vice versa. If this is the case I would have expected the values to be the same for the corresponding pair of tcp_read and tcp_write.
Could someone help me understand the difference in values, and which metric out of tcp_read or tcp_write would be preferable for measuring inbound TCP size to the Nginx workload?