Http: TLS handshake error from tap pod

We are seeing tls handshake failure on the tap pod container and want to know if this is a soft or hard error.

time=“2023-10-27T17:43:00Z” level=info msg=“caches synced”

time=“2023-10-27T17:43:00Z” level=info msg=“starting tap API server on :8089” addr=“:8089” component=tap

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59616: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59584: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59650: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59630: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59716: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59634: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59700: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59706: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59734: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59640: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59746: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59764: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59752: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59738: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59724: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59794: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59780: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59768: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59750: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59742: EOF

2023/10/27 17:43:06 http: TLS handshake error from 100.65.0.56:59808: EOF

Here is our linkerd check output:

linkerd check -o short

linkerd-webhooks-and-apisvc-tls


:bangbang: proxy-injector cert is valid for at least 60 days

certificate will expire on 2023-10-28T18:44:12Z

see https://linkerd.io/2.13/checks/#l5d-proxy-injector-webhook-cert-not-expiring-soon for hints

:bangbang: sp-validator cert is valid for at least 60 days

certificate will expire on 2023-10-28T18:44:12Z

see https://linkerd.io/2.13/checks/#l5d-sp-validator-webhook-cert-not-expiring-soon for hints

:bangbang: policy-validator cert is valid for at least 60 days

certificate will expire on 2023-10-28T18:44:11Z

see https://linkerd.io/2.13/checks/#l5d-policy-validator-webhook-cert-not-expiring-soon for hints

linkerd-version


:bangbang: cli is up-to-date

is running version 2.13.7 but the latest stable version is 2.14.2

see https://linkerd.io/2.13/checks/#l5d-version-cli for hints

control-plane-version


:bangbang: control plane is up-to-date

is running version 2.14.1 but the latest stable version is 2.14.2

see https://linkerd.io/2.13/checks/#l5d-version-control for hints

:bangbang: control plane and cli versions match

control plane running stable-2.14.1 but cli running stable-2.13.7

see https://linkerd.io/2.13/checks/#l5d-version-control for hints

linkerd-control-plane-proxy


:bangbang: control plane proxies are up-to-date

some proxies are not running the current version:

    * linkerd-destination-6cc7bbcbd4-nnbhm (stable-2.14.1)

    * linkerd-identity-56568dd75d-hjdlm (stable-2.14.1)

    * linkerd-proxy-injector-58894667b5-k6wrn (stable-2.14.1)

see https://linkerd.io/2.13/checks/#l5d-cp-proxy-version for hints

:bangbang: control plane proxies and cli versions match

linkerd-destination-6cc7bbcbd4-nnbhm running stable-2.14.1 but cli running stable-2.13.7

see https://linkerd.io/2.13/checks/#l5d-cp-proxy-cli-version for hints

                          linkerd-viz

:bangbang: tap API server cert is valid for at least 60 days

certificate will expire on 2023-10-28T18:44:14Z

see https://linkerd.io/2.13/checks/#l5d-tap-cert-not-expiring-soon for hints

:bangbang: viz extension proxies are up-to-date

some proxies are not running the current version:

    * metrics-api-68cf8f8ffb-swjq4 (stable-2.14.1)

    * prometheus-6988c5c9bf-flxz5 (stable-2.14.1)

    * tap-54545b7748-d6tkt (stable-2.14.1)

    * tap-injector-5c746fc887-9hbll (stable-2.14.1)

    * web-68455945b5-48q9r (stable-2.14.1)

see https://linkerd.io/2.13/checks/#l5d-viz-proxy-cp-version for hints

:bangbang: viz extension proxies and cli versions match

metrics-api-68cf8f8ffb-swjq4 running stable-2.14.1 but cli running stable-2.13.7

see https://linkerd.io/2.13/checks/#l5d-viz-proxy-cli-version for hints

Status check results are √
Linkerd Check Redirection

You probably want to figure out what is running at 100.65.0.56. If that is the IP address of the kubernetes API server, this could indicate that there is a misconfiguration with the tap apiservice (probably related to certificates or cert bundles).

If everything in your system is appears to be working properly–if linkerd check passes, etc`–I generally wouldn’t worry about random connection-oriented logging. If you’re trying to debug a specific problem in your system, however, these log lines can be useful diagnostics.