NGINX sticky session not working with Linkerd

Linkerd General Discussion
1

I installed Linkerd in a fresh GKE cluster using the following Helm Chart: “linkerd-control-plane:1.15.0”
Everything works as expected apart to “session affinity” with nginx which used to work for me before starting using Linkerd.
I want to clarify that Nginx ingress works as expected for everything else, it routes traffic as expected and is part of the mesh as expected.

I even removed the ‘nginx.ingress.kubernetes.io/service-upstream: “true”’ annotation from the ingress itself since I have been told that the ingress needs to communicate with the endpoints directly for the session affinity to work.

Are there any extra special guidelines for it?

Example ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: general-clusterissuer
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/affinity-mode: persistent
    nginx.ingress.kubernetes.io/proxy-body-size: 20m
    nginx.ingress.kubernetes.io/proxy-connect-timeout: '1800'
    nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: '1800'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '1800'
    nginx.ingress.kubernetes.io/session-cookie-expires: '172800'
    nginx.ingress.kubernetes.io/session-cookie-max-age: '172800'
    nginx.ingress.kubernetes.io/session-cookie-name: ses_stickounet
    nginx.ingress.kubernetes.io/session-cookie-path: /
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    nginx.ingress.kubernetes.io/use-regex: 'true'
  name:exp-server-custom-public-endpoints-no-redirect
  namespace: exp-legacy
spec:
  ingressClassName: nginx
  rules:
    - host: dev.exp-ses.us
      http:
        paths:
          - backend:
              service:
                name: exp-server
                port:
                  number: 80
            path: /
            pathType: ImplementationSpecific
  tls:
    - hosts:
        - dev.exp-ses.us
      secretName: dev-exp-us
2

Unfortunately, this is more of a NGINX configuration question than a Linkerd one.

One thing you can do on the Linkerd side would be to look at the linkerd proxy logs to see what ip address NGINX is connecting to and verifying that NGINX is connecting to the service cluster IP instead of to an endpoint IP.

3

Any luck on checking that? :slightly_smiling_face:

4

Couldn’t find any issue with NGINX, without Linkerd everything works perfectly. Unfortunately I had to move to a k8s headless service to force direct endpoint discovery for services I want to use with session affinity and Linkerd

5

Can you tell me exactly how you installed NGINX?

6

NGINX helm chart 4.8.3 (up to date)
With the following extra values for Linkerd

          podAnnotations:
            linkerd.io/inject: enabled
            # This annotation is important as it doesn't route traffic through the proxy which enables ip preservation
            config.linkerd.io/skip-inbound-ports: 80,443 # the workaround
7

OK, thanks, I’ll try this in a bit…

1 Like