# How to reduce the memory usage of filebeat

**URL:** <https://discuss.elastic.co/t/how-to-reduce-the-memory-usage-of-filebeat/283401>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 6, 2021, 5:19am UTC](https://discuss.elastic.co/t/how-to-reduce-the-memory-usage-of-filebeat/283401 "2021-09-06T05:19:22Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\
**Post date:** [September 6, 2021, 5:19am UTC](https://discuss.elastic.co/t/how-to-reduce-the-memory-usage-of-filebeat/283401/1 "2021-09-06T05:19:22Z")

</div>

I am using filebeat to collect log files.

I'm using filebeat to collect log files, and on one of my servers, filebeat's memory usage is high.  
I would like to limit the memory usage of filebeat.

I have set up `queue.mem` by referring to the following page, but the situation is the same as before.  
(I think the default value was used because it was not mentioned before the configuration)

> **[Configure the internal queue | Filebeat Reference \[master\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/master/configuring-internal-queue.html)**

Why does the memory usage not change?  
Also, is there a better way?

```auto
vi /etc/filebeat/filebeat.yml

... snip ...

# ======= configuring internal queue =======
queue.mem:
  events: 128
  flush.min_events: 64
  flush.timeout: 5s

```

```auto
top

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 1851 mysql 20 0 6724m 3.8g 6608 S 7.3 48.3 50:59.88 mysqld
 2229 root 20 0 1075m 57m 23m S 2.7 0.7 0:04.29 filebeat

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 6, 2021, 5:23am UTC](https://discuss.elastic.co/t/how-to-reduce-the-memory-usage-of-filebeat/283401/2 "2021-09-06T05:23:09Z")

</div>

Can you elaborate which memory value you are trying to reduce?

---

<div class="post-metadata">

**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\
**Post date:** [September 6, 2021, 5:26am UTC](https://discuss.elastic.co/t/how-to-reduce-the-memory-usage-of-filebeat/283401/3 "2021-09-06T05:26:38Z")

</div>

These are the values of `VIRT` and `RES` as seen by the `top` command.  
These are currently the second most common values after `mysql`.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 6, 2021, 5:27am UTC](https://discuss.elastic.co/t/how-to-reduce-the-memory-usage-of-filebeat/283401/4 "2021-09-06T05:27:12Z")

</div>

Is it causing issues?  
The `RES` amount is pretty low, and `VIRT` is managed by the OS, not Filebeat.

---

<div class="post-metadata">

**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\
**Post date:** [September 6, 2021, 5:35am UTC](https://discuss.elastic.co/t/how-to-reduce-the-memory-usage-of-filebeat/283401/5 "2021-09-06T05:35:28Z")

</div>

Fortunately, we are not experiencing any problems now.

The server was rebooted early in the morning, and it seems that filebeat was temporarily overloaded.

It seems to have settled down now, but exceeding 1G of virtual memory and 60MB of real memory seems to be a bit of a burden on the server, and we are looking for a way to deal with it.

This may be unrelated to the problem, but there seems to be a large number of processes.

```auto
ps -efL | grep filebeat | wc -l
12

```

The number of logs I am sending to logstash with filebeat is four.  
Is the number of processes supposed to be this large?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 6, 2021, 5:57am UTC](https://discuss.elastic.co/t/how-to-reduce-the-memory-usage-of-filebeat/283401/6 "2021-09-06T05:57:52Z")

</div>

Virtual memory is managed entirely by the OS.

You _might_ be able to minimise its use by tweaking the `close_*` parameters.

---

<div class="post-metadata">

**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\
**Post date:** [September 6, 2021, 6:05am UTC](https://discuss.elastic.co/t/how-to-reduce-the-memory-usage-of-filebeat/283401/7 "2021-09-06T06:05:20Z")

</div>

I am decreasing the values of `events` and `flush.min_events` in steps and observing the values with the `top` command, but the values of `VIRT` and `RES` are not decreasing.

`events` is set from 4096 to 2048, 1024, 512, 256, 128  
The `min_events` has been reduced by 6 steps from 2048 to 1024, 512, 256, 128, 64  
but there is no change in the values.

Do these operations inherently lower the memory value?

Or should I assume that the memory usage is at a reasonable value and will not go any lower?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 4, 2021, 8:05am UTC](https://discuss.elastic.co/t/how-to-reduce-the-memory-usage-of-filebeat/283401/8 "2021-10-04T08:05:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
