Dynamic Request routing across namespaces

Hello everyone,
I have been trying to set up Dynamic Request routing, following the docs at Configuring dynamic request routing

Until I create the frontend and both backends in one namespace, everything works pretty well, but when I try to put backends to separate namespaces, it does not work anymore. It looks like the namespace in HttpRoute backendRefs ignores the namespace definition. Is it possible to route requests dynamically amongst multiple namespaces?

My deployment topology is:
namespace test - frontend-podinfo chart
namespace test-v2 - backend-b-podinfo chart
namespace test-v3 - backend-c-podinfo chart

You can deploy it using this script:

kubectl create ns test --dry-run -o yaml | linkerd inject - | kubectl apply -f -
kubectl create ns test-v2 --dry-run -o yaml | linkerd inject - | kubectl apply -f -
kubectl create ns test-v3 --dry-run -o yaml | linkerd inject - | kubectl apply -f -

helm install frontend -n test --set backend=http://backend-b-podinfo.test-v2.svc.cluster.local:9898/env podinfo/podinfo
helm install backend-b -n test-v2 --set ui.message='B TEST2 backend' podinfo/podinfo
helm install backend-c -n test-v3 --set ui.message='C TEST3 backend' podinfo/podinfo

If a header x-request-id: alternative the request should be routed to backend-c-podinfo

HTTP Route definition:

apiVersion: policy.linkerd.io/v1beta2
kind: HTTPRoute
metadata:
  name: backend-router
  namespace: test-v2
spec:
  parentRefs:
    - name: backend-b-podinfo
      namespace: test-v2
      kind: Service
      group: core
      port: 9898
  rules:
    - matches:
      - headers:
        - name: "x-request-id"
          value: "alternative"
      backendRefs:
        - name: "backend-c-podinfo"
          namespace: "test-v3"
          port: 9898
    - backendRefs:
      - name: "backend-b-podinfo"
        namespace: "test-v2"
        port: 9898

Using port-forwarding to frontend service I get this error:

curl -sX POST -H 'x-request-id: alternative' localhost:9898/echo                                                                                                                                                             <aws:nka_dev>
[
  "backend http://backend-b-podinfo.test-v2.svc.cluster.local:9898/env response status code 500"
]%  

The error is logged out in frontend pod, in linkerd-proxy container:

[ 3478.322077s] INFO ThreadId(01) outbound:proxy{addr=10.100.51.255:9898}:rescue{client.addr=172.31.65.219:49976}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=logical service 10.100.51.255:9898: HTTP request configured to fail with 500 Internal Server Error: Service not found backend-c-podinfo error.sources=[HTTP request configured to fail with 500 Internal Server Error: Service not found backend-c-podinfo]

In the real world, we want to have a single service in default namespace (similar to frontend pod), that will insert the control header into the request and 2 (or more) versions of our system (backends in the example), each in its own namespace and dynamically route between them, so some of the clients get newer version, rest gets the old one.

Is there any way how to achieve this? How can I route between namespaces using HttpRoute?

Hi @jdolan

I looked into this and it is, indeed, a bug. Thanks for the report!

Fix being developed here: Use namespace field of backend ref when it is set by adleong · Pull Request #10909 · linkerd/linkerd2 · GitHub