/debug/pprof/heap`. It'd be nice if you could send one to us when the memory increases so we can take a look. If you want you can directly explore it with `go tool pprof`.
By the way, what memory limit did you set to these pods? Did you change filebeat version last week?
---
### Author:  [@TimWard](https://discuss.elastic.co/u/TimWard)
#### Post date: [June 25, 2018, 3:24pm UTC](https://discuss.elastic.co/t/filebeat-memory-leak-on-kubernetes/137208/5 "2018-06-25T15:24:05Z")
Profile output will follow shortly. The memory configuration is taken directly from the published Kubernetes deployment file (I just copied it without trying to understand it):
```
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
```
The sequence of upgrades was as follows:
- I installed Filebeat 6.2.4 a couple of weeks ago. As far as I know it was initially working fine.
- At some point last Friday it started misbehaving, with pods restarted frequently due to being out of memory.
- As I knew there was a memory leak issue that was fixed in 6.3.0 I installed that, on Friday, to see if it would make any difference, and it appeared not to.
- I'm still running 6.3.0, and it still appears to be misbehaving.
---
### Author:  [@jsoriano](https://discuss.elastic.co/u/jsoriano)
#### Post date: [June 25, 2018, 3:28pm UTC](https://discuss.elastic.co/t/filebeat-memory-leak-on-kubernetes/137208/6 "2018-06-25T15:28:20Z")
While we find what can be happening, could you try to increase the memory limit?
---
### Author:  [@TimWard](https://discuss.elastic.co/u/TimWard)
#### Post date: [June 25, 2018, 3:42pm UTC](https://discuss.elastic.co/t/filebeat-memory-leak-on-kubernetes/137208/7 "2018-06-25T15:42:53Z")
What's the best way of getting 4,100 lines of profile to you - it'll only let me attach image files?
This suggests that memory was growing up to the point at which I took the profile:
```
[root@filebeat-deployment-7d8b947947-kv48z filebeat]# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 1.0 0.6 626728 105964 ? Ssl 15:16 0:11 filebeat -c /etc/filebeat.yml -e --httpprof :8808
root 18 0.0 0.0 11832 2916 ? Ss 15:34 0:00 /bin/bash
root 33 0.0 0.0 51720 3456 ? R+ 15:34 0:00 ps -aux
[root@filebeat-deployment-7d8b947947-kv48z filebeat]# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 1.0 0.6 626728 108852 ? Ssl 15:16 0:12 filebeat -c /etc/filebeat.yml -e --httpprof :8808
root 18 0.0 0.0 11832 2916 ? Ss 15:34 0:00 /bin/bash
root 34 0.0 0.0 51720 3444 ? R+ 15:35 0:00 ps -aux
[root@filebeat-deployment-7d8b947947-kv48z filebeat]# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 1.0 0.7 638092 116572 ? Ssl 15:16 0:13 filebeat -c /etc/filebeat.yml -e --httpprof :8808
root 18 0.0 0.0 11832 2916 ? Ss 15:34 0:00 /bin/bash
root 36 0.0 0.0 51720 3440 ? R+ 15:37 0:00 ps -aux
[root@filebeat-deployment-7d8b947947-kv48z filebeat]# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 1.0 0.7 645748 120616 ? Ssl 15:16 0:14 filebeat -c /etc/filebeat.yml -e --httpprof :8808
root 18 0.0 0.0 11832 2916 ? Ss 15:34 0:00 /bin/bash
root 37 0.0 0.0 51720 3448 ? R+ 15:38 0:00 ps -aux
[root@filebeat-deployment-7d8b947947-kv48z filebeat]# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 1.0 0.7 648216 124672 ? Ssl 15:16 0:14 filebeat -c /etc/filebeat.yml -e --httpprof :8808
root 18 0.0 0.0 11832 2916 ? Ss 15:34 0:00 /bin/bash
root 38 0.0 0.0 51720 3308 ? R+ 15:39 0:00 ps -aux
[root@filebeat-deployment-7d8b947947-kv48z filebeat]#
```
I don't really want to try increasing the limit at the moment because there's something really sick with our K8s cluster, and other people think that the Filebeat problems might be the cause, or at least a contributory factor, and they'd rather like me to stop running it for the time being.
---
### Author:  [@TimWard](https://discuss.elastic.co/u/TimWard)
#### Post date: [June 28, 2018, 8:04am UTC](https://discuss.elastic.co/t/filebeat-memory-leak-on-kubernetes/137208/8 "2018-06-28T08:04:00Z")
We had other reasons to rebuild the K8s cluster. On restarting (and with Metricbeat now running as well) the Filebeat memory usage was flat for a couple of hours and then started increasing again - one pod restarted overnight.
---
### Author:  [@exekias](https://discuss.elastic.co/u/exekias)
#### Post date: [June 28, 2018, 8:49am UTC](https://discuss.elastic.co/t/filebeat-memory-leak-on-kubernetes/137208/9 "2018-06-28T08:49:42Z")
Hi @TimWard,
Do you see any errors in Filebeat logs?
Best regards
---
### Author:  [@TimWard](https://discuss.elastic.co/u/TimWard)
#### Post date: [July 2, 2018, 11:08am UTC](https://discuss.elastic.co/t/filebeat-memory-leak-on-kubernetes/137208/10 "2018-07-02T11:08:13Z")
Not consistently (there were some apparently intermittent DNS resolution errors for some periods, but not others).
---
### Author:  [@TimWard](https://discuss.elastic.co/u/TimWard)
#### Post date: [July 9, 2018, 10:35am UTC](https://discuss.elastic.co/t/filebeat-memory-leak-on-kubernetes/137208/11 "2018-07-09T10:35:47Z")
We have increased the Filebeat memory limit to 500M and it now only gets OOM killed every couple of days rather than every couple of hours.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 6, 2018, 10:35am UTC](https://discuss.elastic.co/t/filebeat-memory-leak-on-kubernetes/137208/12 "2018-08-06T10:35:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.