will
April 22, 2023, 12:52am
1
Hello - great to see this forum, nice idea.
To keep things simple - I was just looking for an update on egress traffic control. From what I’ve pieced together:
It was originally planned for 2022 but the gateway api adoption delayed some things
As of last year it was looking like 2.13ish
Are you thinking this will be coming in the next release or two?
Is this the appropriate GitHub issue to keep an eye on?
opened 06:58PM - 07 Jun 21 UTC
help wanted
pinned
needs/design
## Feature Request
Enable Linkerd policies to fully specify the external reso… urces a cluster can access, a la [egress gateways](https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway/#egress-gateway-for-http-traffic) from Istio.
### What problem are you trying to solve?
In high compliance environments, it's useful to be able to use the service mesh as part of the network security of a cluster.
Suppose I wish to have some pods interact with the GitHub API. We'll call this application our `github-agent`. Without using any third party tooling, those pods need port 443 outbound to be open. However, if a supply chain attack injects code into my cluster, they could phone home to any command and control (C2) node on port 443 to perform malicious acts.
A default deny policy in a namespace ensures that outbound network traffic must be authorized, but K8s network policies are layer 3 (IP address and protocol) based, and cannot enforce HTTPS or mitigate supply chain attacks such as Magecart.
A layer 7 proxy solution is needed to ensure our `github-agent` application can only access `https://api.github.com` with a valid certificate.
### How should the problem be solved?
Either of the two methods seems sound to me:
1. Client-side egress filtering
Client-side profiles have an allowlist mode which acts as an additional layer of filtering that cannot be circumvented by application code (perhaps via the CNI Plugin and iptables rules?).
Pros: the compute and networking overhead here is already paid for by existing sidecar containers.
Cons: the NetworkPolicy that targets those pods will likely still need to allow port 443 egress to 0.0.0.0/0. (Is there an alternative?).
2. Namespace or cluster-wide egress gateway
A cluster-wide egress gateway is deployed by Linkerd which proxies requests to external services.
Pros: Only that egress gateway needs port 443 egress.
Cons: contrary to some of the goals of Linkerd, this creates an additional network hop and centralizes network traffic in the egress gateway(s)
### Any alternatives you've considered?
In clusters I manage, I use a default deny to block all applications in the cluster from making external network requests. Then, I deploy:
* A `Service` `github-proxy` which targets the pods managed by the same named deployment outbound...
* A `NetworkPolicy` `github-proxy-egress` which allows the port 443 egress and certain labeled pods port 80 ingress to...
* A `Deployment` `github-proxy` running nginx with a specific configuration:
http {
upstream api.github.com {
server api.github.com:443 max_fails=0;
}
server {
# ...
location / {
# ...
proxy_pass https://api.github.com/;
}
}
}
This is fairly high overhead, requiring deploying 3 Kubernetes resources and modifying application code. The latter especially isn't always possible. If `github-agent` were a third party application, it may hardcode requests and not be configurable to use a proxy.
### How would users interact with this feature?
Knobs/controls I would expect on this resource for each desired hostname:
* Optional ports and protocols (HTTP, HTTP/2, gRPC, and opaque TCP or UDP)
* An optional flag to control whether this is an allowlist (Linkerd will close connections to not explicitly allowed hosts).
* An optional flag to control whether TLS is enforced, for non-opaque protocols.
* An optional flag to control whether telemetry is enabled, for non-opaque protocols
* An optional field to determine what trace header propagation format to use, for future proofing. This ought to be `None` by default, to drop trace headers on outbound requests to avoid leaking information.) For cross-cluster communication, I would imagine this supporting different propagation formats in the future.
My opinions on defaults:
* Protocol detection should be enabled, but it should be possible to define the protocol to constrain traffic, e.g.: I know my destination API only supports gRPC, do not allow general HTTPS traffic.
* A non-matching hostname will be proxied as though it had default settings
* TLS enforcement is opt-out unless the protocol is set to HTTP or opaque
* Telemetry should be opt-out for supported protocols
* Trace propagation should be opt-in to avoid leaking application details by default
The defaults therefore would result in Linkerd acting as a MITM for external resources and collecting telemetry, dropping trace headers on outbound requests.
For users with compliance needs, setting the allowlist flag would cause the sidecar to close connections to non-matching hostnames and protocols by default.
Sorry to ask in another location but I just hadn’t seen an update in awhile. We are eagerly awaiting this functionality.
Hey Will. We’re planning out the post-2.13 right now and this is on the list of possible things to tackle. You are definitely not the only one asking. Stay tuned…
1 Like
will
June 15, 2023, 5:24pm
3
Hey - just checking in - did the post-2.13 roadmap firm up? Thanks!
Howdy. Yes, 2.14 is firmer now but unfortunately egress did not make the cut. We are aggressively targeting 2.14 for August and it will be a candidate for 2.15. Sorry to be the bearer of bad news.
will
August 23, 2023, 8:26pm
5
Hey - congrats on 2.14. The flat networking changes will help quite a bit. I hope egress filtering isn’t something you’re slating for Linkerd Enterprise ?
I’m not entirely sure yet which piece(s) will end up in enterprise, if any. You can see the current list of Linkerd Enterprise features here FWIW.