# Potential (slow) memory leak for linkerd-proxy sidecar for single workload on 2.13.3

**URL:** <https://linkerd.buoyant.io/t/potential-slow-memory-leak-for-linkerd-proxy-sidecar-for-single-workload-on-2-13-3/148>\
**Category:** Linkerd General Discussion\
**Created:** [June 1, 2023, 5:03pm UTC](https://linkerd.buoyant.io/t/potential-slow-memory-leak-for-linkerd-proxy-sidecar-for-single-workload-on-2-13-3/148 "2023-06-01T17:03:24Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![neal](https://yyz1.discourse-cdn.com/flex029/user_avatar/linkerd.buoyant.io/neal/32/54_2.png) [@neal](https://linkerd.buoyant.io/u/neal)\
**Post date:** [June 1, 2023, 5:03pm UTC](https://linkerd.buoyant.io/t/potential-slow-memory-leak-for-linkerd-proxy-sidecar-for-single-workload-on-2-13-3/148/1 "2023-06-01T17:03:25Z")

</div>

Hello! I’ve noticed that we have a single deployment for which the linkerd-proxy container appears to have a slow memory leak. It currently has a memory limit of 500Mi for the container, and will take over a week to hit that limit, but memory usage for it really stands out among all of our other workloads.

I know this is probably nearly impossible to diagnose with the information I’ve given, but are there any properties of a workload that might interact with linkerd to precipitate something like this? As far as I know, we’re not having any operational problems with the workload. It’s meshed, subject to authz policies, and working fine.

I’ve attached a screenshot that shows memory usage for the linkerd-proxy container for all of our deployments. You can see that only a single one (out of dozens) shows this pattern.

 ![Screenshot 2023-06-01 at 9.58.45 AM](https://canada1.discourse-cdn.com/flex029/uploads/buoyant1/original/1X/fcea05d69abf758a152ba20b1ed9e1dcf36fc3df.png)

---

<div class="post-metadata">

**Author:** ![william](https://yyz1.discourse-cdn.com/flex029/user_avatar/linkerd.buoyant.io/william/32/5_2.png) [@william](https://linkerd.buoyant.io/u/william)\
**Post date:** [June 2, 2023, 1:43pm UTC](https://linkerd.buoyant.io/t/potential-slow-memory-leak-for-linkerd-proxy-sidecar-for-single-workload-on-2-13-3/148/2 "2023-06-02T13:43:12Z")

</div>

Which version is the proxy?

Edit: never mind, just saw that in the title. Interesting. Can you connect your deployment to Buoyant Cloud and send us a diagnostic bundle? That would be the quickest way to dig into this.

---

<div class="post-metadata">

**Author:** ![neal](https://yyz1.discourse-cdn.com/flex029/user_avatar/linkerd.buoyant.io/neal/32/54_2.png) [@neal](https://linkerd.buoyant.io/u/neal)\
**Post date:** [June 2, 2023, 11:54pm UTC](https://linkerd.buoyant.io/t/potential-slow-memory-leak-for-linkerd-proxy-sidecar-for-single-workload-on-2-13-3/148/3 "2023-06-02T23:54:21Z")

</div>

I pressed the button to send diagnostics. I was told I’d get an email, but I’m not sure that ever happened.

---

<div class="post-metadata">

**Author:** ![william](https://yyz1.discourse-cdn.com/flex029/user_avatar/linkerd.buoyant.io/william/32/5_2.png) [@william](https://linkerd.buoyant.io/u/william)\
**Post date:** [June 13, 2023, 10:42pm UTC](https://linkerd.buoyant.io/t/potential-slow-memory-leak-for-linkerd-proxy-sidecar-for-single-workload-on-2-13-3/148/4 "2023-06-13T22:42:23Z")

</div>

Neal, I just wanted to follow up on this. Did you ever get an email with the diagnostic ID in it?

---

<div class="post-metadata">

**Author:** ![neal](https://yyz1.discourse-cdn.com/flex029/user_avatar/linkerd.buoyant.io/neal/32/54_2.png) [@neal](https://linkerd.buoyant.io/u/neal)\
**Post date:** [June 15, 2023, 5:16pm UTC](https://linkerd.buoyant.io/t/potential-slow-memory-leak-for-linkerd-proxy-sidecar-for-single-workload-on-2-13-3/148/5 "2023-06-15T17:16:58Z")

</div>

Hello! I don’t think I ever got an email.

---

<div class="post-metadata">

**Author:** ![william](https://yyz1.discourse-cdn.com/flex029/user_avatar/linkerd.buoyant.io/william/32/5_2.png) [@william](https://linkerd.buoyant.io/u/william)\
**Post date:** [June 15, 2023, 6:48pm UTC](https://linkerd.buoyant.io/t/potential-slow-memory-leak-for-linkerd-proxy-sidecar-for-single-workload-on-2-13-3/148/6 "2023-06-15T18:48:04Z")

</div>

Interesting. I looked through the send logs and also don’t see anything. Can you try again? And note the timestamp?

---

<div class="post-metadata">

**Author:** ![neal](https://yyz1.discourse-cdn.com/flex029/user_avatar/linkerd.buoyant.io/neal/32/54_2.png) [@neal](https://linkerd.buoyant.io/u/neal)\
**Post date:** [June 16, 2023, 12:27am UTC](https://linkerd.buoyant.io/t/potential-slow-memory-leak-for-linkerd-proxy-sidecar-for-single-workload-on-2-13-3/148/7 "2023-06-16T00:27:13Z")

</div>

I just tried again at 5:25pm PDT. I didn’t receive a timestamp form Buoyant or anything, in case that’s what you mean by timestamp.

 ![Screenshot 2023-06-15 at 5.25.29 PM](https://canada1.discourse-cdn.com/flex029/uploads/buoyant1/original/1X/4c67bb1d854c8c3460684fe5e691147dcccd1a2d.png)

---

<div class="post-metadata">

**Author:** ![william](https://yyz1.discourse-cdn.com/flex029/user_avatar/linkerd.buoyant.io/william/32/5_2.png) [@william](https://linkerd.buoyant.io/u/william)\
**Post date:** [July 1, 2023, 2:25pm UTC](https://linkerd.buoyant.io/t/potential-slow-memory-leak-for-linkerd-proxy-sidecar-for-single-workload-on-2-13-3/148/8 "2023-07-01T14:25:38Z")

</div>

Neal, I dropped the ball on replying to this. I trawled our outbound email logs and also don’t see any email sent to you at that time. Not sure why that would be. I will file an internal ticket for us to investigate. Sorry about that!

In terms of that one workload, can you describe the nature of the traffic a bit? What does it do that’s different from the other meshed workloads?

---

<div class="post-metadata">

**Author:** ![neal](https://yyz1.discourse-cdn.com/flex029/user_avatar/linkerd.buoyant.io/neal/32/54_2.png) [@neal](https://linkerd.buoyant.io/u/neal)\
**Post date:** [July 3, 2023, 11:08pm UTC](https://linkerd.buoyant.io/t/potential-slow-memory-leak-for-linkerd-proxy-sidecar-for-single-workload-on-2-13-3/148/9 "2023-07-03T23:08:57Z")

</div>

The workload takes requests which are large json blobs (mostly base64 encoded images), and returns json responses which are scan results of those images. We have other workloads which are similar, and not experiencing a memory leak. I’m not sure what it is about this workload that distinguishes it from other meshed workloads. 😕

At this point, we at least have a work-around: the ol’ reboot it daily. (The leak would probably take about 4-5 days to cause an OOM, so this is more aggressive than is necessary, but fine.)
