Linkerd and TCP connections not working.

Linkerd General Discussion
1

Hello folks
I have multiple microservices.
They all communicate with TCP, and some of them also have HTTP.
I’ve injected the Linkerd into them, and now nothing is working!
I’ve searched and realized I should add the config.linkerd.io/opaque-ports annotations to Deployments and Services.
Added and did not work again.
Here is an example (Other Microservices are like this user example):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: users
  name: users
  namespace: staging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: users
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
        config.linkerd.io/opaque-ports: "3002"
      labels:
        app: users
      name: users-pod
    spec:
      containers:
      - image: MY-REPO/users:f2e2c
        name: users-container
        ports:
        - containerPort: 3002
          protocol: TCP

apiVersion: v1
kind: Service
metadata:
  annotations:
    config.linkerd.io/opaque-ports: "3002"
    ###
    # I have also tried the svc port: `config.linkerd.io/opaque-ports: "5000"`, didn't work either
   ###
  name: users-service
  namespace: staging
spec:
  ports:
  - name: ws
    port: 5000
    protocol: TCP
    targetPort: 3002
  selector:
    app: users
  type: ClusterIP

Logs seemed ok:

✗ k -n staging logs users-c7d55c786-x6b7l
Defaulted container "linkerd-proxy" out of: linkerd-proxy, users-container, linkerd-init (init)
[     0.009122s]  INFO ThreadId(01) linkerd2_proxy: release 2.210.4 (5a910be) by linkerd on 2023-11-22T17:01:46Z
[     0.012860s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.015042s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.015060s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.015064s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.015067s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.015070s]  INFO ThreadId(01) linkerd2_proxy: Local identity is default.staging.serviceaccount.identity.linkerd.cluster.local
[     0.015073s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.015077s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.039768s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=default.staging.serviceaccount.identity.linkerd.cluster.local

Apps, cannot talk to gather (Timeout), but as soon as I disabled Linkerd, everything got back to being OK.