NGINX sticky session not working with Linkerd

Linkerd General Discussion
1

I installed Linkerd in a fresh GKE cluster using the following Helm Chart: “linkerd-control-plane:1.15.0”
Everything works as expected apart to “session affinity” with nginx which used to work for me before starting using Linkerd.
I want to clarify that Nginx ingress works as expected for everything else, it routes traffic as expected and is part of the mesh as expected.

I even removed the ‘nginx.ingress.kubernetes.io/service-upstream: “true”’ annotation from the ingress itself since I have been told that the ingress needs to communicate with the endpoints directly for the session affinity to work.

Are there any extra special guidelines for it?

Example ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: general-clusterissuer
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/affinity-mode: persistent
    nginx.ingress.kubernetes.io/proxy-body-size: 20m
    nginx.ingress.kubernetes.io/proxy-connect-timeout: '1800'
    nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: '1800'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '1800'
    nginx.ingress.kubernetes.io/session-cookie-expires: '172800'
    nginx.ingress.kubernetes.io/session-cookie-max-age: '172800'
    nginx.ingress.kubernetes.io/session-cookie-name: ses_stickounet
    nginx.ingress.kubernetes.io/session-cookie-path: /
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    nginx.ingress.kubernetes.io/use-regex: 'true'
  name:exp-server-custom-public-endpoints-no-redirect
  namespace: exp-legacy
spec:
  ingressClassName: nginx
  rules:
    - host: dev.exp-ses.us
      http:
        paths:
          - backend:
              service:
                name: exp-server
                port:
                  number: 80
            path: /
            pathType: ImplementationSpecific
  tls:
    - hosts:
        - dev.exp-ses.us
      secretName: dev-exp-us
2

Unfortunately, this is more of a NGINX configuration question than a Linkerd one.

One thing you can do on the Linkerd side would be to look at the linkerd proxy logs to see what ip address NGINX is connecting to and verifying that NGINX is connecting to the service cluster IP instead of to an endpoint IP.

3

Any luck on checking that? :slightly_smiling_face:

4

Couldn’t find any issue with NGINX, without Linkerd everything works perfectly. Unfortunately I had to move to a k8s headless service to force direct endpoint discovery for services I want to use with session affinity and Linkerd

5

Can you tell me exactly how you installed NGINX?

6

NGINX helm chart 4.8.3 (up to date)
With the following extra values for Linkerd

          podAnnotations:
            linkerd.io/inject: enabled
            # This annotation is important as it doesn't route traffic through the proxy which enables ip preservation
            config.linkerd.io/skip-inbound-ports: 80,443 # the workaround
7

OK, thanks, I’ll try this in a bit…

1 Like
8

Hey @Flynn and @arielzadino isn’t this stopping nginx from sending traffic to the injected linkerd proxy for ports 80 and 443? In theory this sounds a workaround if you don’t allow unmeshed traffic with authorization policy this should be an issue!

I think if there is a way the mtls cert from the NGINX ingress to be generated by cert manager if it is used by Linkerd to be a better workaround as this way nginx will authenticate to backend pods. Only metric collections could be impacted but Nginx Plus should handle this.

9

@Alex By the way why is this needed when I tested and everything seems fine when Nginx talks directly to the pods and mTLS is working as well.

10

Ok found it Ingress traffic | Linkerd and it is not clear reason as if the ingress can do retransmissions for example then it will work fine.

11

Hey @Nikolayy1

The behavior here will be slightly different depending on if nginx is connecting to the service cluster IP or to the service endpoints directly.

If nginx connects to the service cluster IP then it will only see one IP address and Linkerd will handle load balancing. This means that nginx sticky sessions will not work.

On the other hand, if nginx connects to the service endpoints directly then it is managing its own connections and load balancing and can perform sticky sessions. However, this means that any Linkerd policy which is configured on the Service resource (such as outbound HTTPRoutes or ServiceProfiles) will not apply.

In either case, the connection will be fully mTLS. Does this answer your question?

1 Like
12

(I’ll take @Alex‘s answer a bit further: if you want sticky sessions you must configure your ingress controller to use endpoint IPs, which means that you must also have your ingress controller manage retries, timeouts, etc., because Linkerd can’t do it. This is true irrespective of which ingress controller you’re using. :slight_smile: )