Custom headers being stripped before reaching application - is this expected?

Linkerd General Discussion
,
1

I’ve been debugging why custom HTTP headers aren’t reaching our application in a Linkerd-injected environment, and I’ve narrowed it down to the proxy layer. I’m wondering if this is expected behavior or if there’s a configuration I’m missing.

Setup

  • Linkerd edge-25.9.1 on AWS EKS
  • Simple traffic path: ALB → Linkerd proxy → nginx → PHP app
  • Server resource with proxyProtocol: HTTP/1 and accessPolicy: deny with an AuthorizationPolicy allowing all traffic

What’s happening

When I send custom headers like X-Force-Tracking: 1 or X-Test-Debug-Header: test, they never reach nginx or the application. But AWS infrastructure headers like X-Amzn-Trace-Id come through fine.

I added logging to nginx to see what it receives:

{
“x_force_tracking”: “”,              // empty
“x_test_debug_header”: “”,            // empty
“x_amzn_trace_id”: “Root=1-6904…”  // present
}

So the headers are being stripped somewhere between the ALB and nginx.

What I’ve tested

I have an identical environment without Linkerd injection where the same headers work perfectly. Same ALB config, same nginx config, same app code. The only difference is the Linkerd injection.

I tried:

  • Different header names (tested 6 variations including ones without the X- prefix)
  • Changing proxyProtocol from HTTP/1 to unknown
  • Changing accessPolicy from deny to all-unauthenticated
  • Verified ALB has drop_invalid_header_fields=true on both environments

None of that made any difference. Custom headers are consistently stripped while infrastructure headers pass through.

What I’m trying to do

We need to implement debug tracing where developers can send a custom header to trigger detailed logging/tracing on specific requests. Pretty standard debugging pattern but it doesn’t work with Linkerd in the mix.

Questions

Is this expected behavior? Is there some proxy configuration or annotation that controls which headers get passed through? I couldn’t find anything in the docs about this.

If this is by design, what’s the recommended workaround? Query parameters? Cookies? Or is there a way to whitelist certain header patterns?

Any help appreciated. I’ve spent about 7 hours systematically testing this and I’m confident the stripping is happening in the proxy, but I can’t figure out why or how to control it.

  ---
  Environment details if needed:
  apiVersion: policy.linkerd.io/v1beta3
  kind: Server
  spec:
    podSelector:
      matchLabels:
        app: raven
    port: 8080
    proxyProtocol: HTTP/1
    accessPolicy: deny

  ---
2

That is surprising to me…. in part because I have just been updating some older docs that state that Linkerd will generally propagate unknown headers to allow for distributed tracing. Maybe @Flynn has some ideas?

3

This is very strange – I regularly demo using a custom “x-faces-user” header for A/B testing with an HTTPRoute, and it works fine (I just doublechecked it with edge-25.9.4 and edge-25.10.7). The architecture here is:

web browser → faces-gui → face → smiley

with everything other than the browser in the mesh, and with faces-gui acting as the ingress. There’s an HTTPRoute doing header-based routing on x-faces-user for the smiley workload, so the header has to be propagated multiple times.

To state the obvious, I’m using neither an ALB nor nginx, but I feel confident in ruling out Linkerd. I actually don’t know of a way to even force Linkerd to strip headers… :thinking:

4

Thank you both!

I had a feeling it was something I was doing, or not doing. So this confirmation helps me dig further into what Ive got configured wrong.

Ill update with what I find and what the ultimate fix is, just incase someone else faces similar issues later.