# Filebeat Consume High Amount of Memory

**URL:** <https://discuss.elastic.co/t/filebeat-consume-high-amount-of-memory/152929>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 18, 2018, 3:45am UTC](https://discuss.elastic.co/t/filebeat-consume-high-amount-of-memory/152929 "2018-10-18T03:45:36Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![fiq](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fiq](https://discuss.elastic.co/u/fiq)\
**Post date:** [October 18, 2018, 3:45am UTC](https://discuss.elastic.co/t/filebeat-consume-high-amount-of-memory/152929/1 "2018-10-18T03:45:36Z")

</div>

Hi,

Filebeat on one of our machine is consuming high amount of memory.  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/c/ec03f143e0079451c092f937031bc32b39e9690c.png)  
Our filebeat configuration is like this below :

```
- type: log
  paths:
  - /var/log/fmw/app/portal/portal-api.log
  include_lines: ['^\d{4}-\d{2}-\d{2}']
  close_inactive: 24h
  close_removed: true
  clean_inactive: 26h
  clean_removed: true
  scan_frequency: 1s
  ignore_older: 25h
  multiline.pattern: '^\d{4}-\d{2}-\d{2}'
  multiline.negate: true
  multiline.match: after
  timeout: 5m
  fields:
    log_type : mobile_portal_api
  fields_under_root : true

```

Filebeat is tracking 13 log file in this server with about 30 million of document sent daily.

Log from filebeat is :

```
2018-10-17T01:06:57.163+0700	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":9605870,"time":9605871},"total":{"ticks":77359460,"time":77359467,"value":77359460},"user":{"ticks":67753590,"time":67753596}},"info":{"ephemeral_id":"65d63e03-a732-473e-a520-2cb14f2cde0e","uptime":{"ms":1343130030}},"memstats":{"gc_next":18316201328,"memory_alloc":13287472672,"memory_total":9897155880544,"rss":3403776}},"filebeat":{"events":{"active":101,"added":2057,"done":1956},"harvester":{"open_files":689,"running":689,"started":14}},"libbeat":{"config":{"module":{"running":13},"reloads":3},"output":{"events":{"acked":1998,"active":69,"batches":30,"total":2067},"read":{"bytes":180},"write":{"bytes":123612873}},"pipeline":{"clients":13,"events":{"active":215,"filtered":14,"published":2043,"total":2057},"queue":{"acked":1883}}},"registrar":{"states":{"current":687,"update":1895},"writes":36},"system":{"load":{"1":2.16,"15":0.59,"5":1.14,"norm":{"1":0.36,"15":0.0983,"5":0.19}}}}}}
2018-10-17T01:06:58.459+0700	INFO	log/harvester.go:216	Harvester started for file: /var/log/fmw/app/credo-integration/credo-integration.log
2018-10-17T01:06:58.465+0700	INFO	log/harvester.go:233	File was removed: /var/log/fmw/app/credo-integration/credo-integration.log. Closing because close_removed is enabled.
2018-10-17T01:06:59.460+0700	INFO	log/harvester.go:216	Harvester started for file: /var/log/fmw/app/credo-integration/credo-integration.log

```

I restarted Filebeat service but the same problem happened again.  
Please give me advice on how to avoid this.

Thanks

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [October 18, 2018, 10:26pm UTC](https://discuss.elastic.co/t/filebeat-consume-high-amount-of-memory/152929/2 "2018-10-18T22:26:08Z")

</div>

From metrics filebeat is processing 689 files.

How big are events in these files? Do you have any long stack-traces?

I see you have 13 modules enabled? Can you share your complete filebeat logs?

The RSS is ~3MB, yet the next GC limit is at ~18GB. This looks like you have seen some very large events from a many files at about the same time.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 15, 2018, 10:26pm UTC](https://discuss.elastic.co/t/filebeat-consume-high-amount-of-memory/152929/3 "2018-11-15T22:26:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.