We have been noticing for quite some time that we have applications that still try to finish work and make new TCP connections after receiving a SIG_TERM.
According to the documentation below, the Linkerd proxy only allows existing connections when it receives the SIG_TERM
Questions:
Does linkerd-await prevent the proxy from doing its SIG_TERM process and not handlind new connections?
config.alpha.linkerd.io/proxy-wait-before-exit-seconds annotation seems to indicate the proxy will wait before shutting down but not about its SIG_TERM process of accepting new connections
config.linkerd.io/shutdown-grace-period is a bit confusing also, this is the timeout before forceful shutdown which should default to 120 but during this grace period, no new connections are accepted?
linkerd-await does not prevent the proxy from receiving SIG_TERM. In the context of shutdown, linkerd-await is used to signal to the proxy when the main container has completed and the proxy should shutdown. This is useful for workloads like Jobs and CronJobs where the main container is expected to complete.
proxy-wait-before-exit-seconds uses a preStop hook to delay the proxy from receiving the SIG_TERM. This allows the main container to receive the SIG_TERM first and do some cleanup before the proxy receives the SIG_TERM and enters graceful shutdown where it stops accepting new connections.
shutdown-grace-period limits the amount of time the proxy can spend in the graceful shutdown state. During this state, the proxy does not accept new connections but will continue to process any connections which have already been established. Any connections still open at the end of the grace period will then be forcefully terminated.
If you have an application that needs to establish new TCP connections after receiving SIG_TERM, config.alpha.linkerd.io/proxy-wait-before-exit-seconds is probably the setting that you want, which delays the proxy from entering the graceful shutdown state and will allow your application some time to establish new connections.
Thank you for clarifying that, its very helpful. A couple of follow ups.
If using linkerd-await --shutdown then the proxy still receives the SIG_TERM right away and stops accepting new connections from the main container? Wanted to clarify this so I can rule this out as an option.
proxy-wait-before-exit-seconds is a pre-stop hook, so my overall terminationGracePeriodSeconds still needs to accomodate the both the container time to stop and the proxy time to shutdown after it receives SIG_TERM?