Memory usage is at extreme high level

alexus · September 4, 2019, 5:35pm

Hello World!

Memory usage seems a bit excessive for metricbeat, is it not? I

# systemctl status metricbeat.service 
● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
   Loaded: loaded (/lib/systemd/system/metricbeat.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-09-04 16:27:07 UTC; 6h ago
     Docs: https://www.elastic.co/products/beats/metricbeat
 Main PID: 30807 (metricbeat)
    Tasks: 50 (limit: 4915)
   Memory: 4.7G
      CPU: 19h 9min 10.908s
   CGroup: /system.slice/metricbeat.service
           └─30807 /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -path.home /usr/share/metricbeat -path.config /etc/metricbeat -path.data /var/lib/metricbeat -path.logs /var/log/metricbeat

Sep 04 16:27:07 app11 systemd[1]: Started Metricbeat is a lightweight shipper for metrics..
# systemctl status filebeat.service 
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-09-03 06:08:46 UTC; 1 day 11h ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 8464 (filebeat)
    Tasks: 25 (limit: 4915)
   Memory: 22.0M
      CPU: 7min 1.198s
   CGroup: /system.slice/filebeat.service
           └─8464 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

Sep 03 06:08:46 app11 systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
#

another system similar to first...

# systemctl status metricbeat.service 
● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
   Loaded: loaded (/lib/systemd/system/metricbeat.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-09-02 21:21:21 UTC; 1 day 20h ago
     Docs: https://www.elastic.co/products/beats/metricbeat
 Main PID: 14947 (metricbeat)
    Tasks: 31 (limit: 4915)
   Memory: 97.4M
      CPU: 12h 52min 42.526s
   CGroup: /system.slice/metricbeat.service
           └─14947 /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -path.home /usr/share/metricbeat -path.config /etc/metricbeat -path.data /var/lib/metricbeat -path.logs /var/log/metricbeat

Sep 02 21:21:21 app12 systemd[1]: Started Metricbeat is a lightweight shipper for metrics..
# systemctl status filebeat.service 
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-09-02 21:21:14 UTC; 1 day 20h ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 14721 (filebeat)
    Tasks: 26 (limit: 4915)
   Memory: 17.1M
      CPU: 1min 29.320s
   CGroup: /system.slice/filebeat.service
           └─14721 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

Sep 02 21:21:14 app12 systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
#

Please advise.

Michael_Madden · September 4, 2019, 6:33pm

Hello,

Thanks for reaching about memory usage of metricbeat. Which version of metricbeat are you running? I found a similar post that details memory issues on 7.1.x releases. The issue in the original post looks like it was fixed with the 7.2.x release.

alexus · September 4, 2019, 10:35pm

# metricbeat version
metricbeat version 6.8.2 (amd64), libbeat 6.8.2 [0ffbeab5a52fa93586e4178becf1252e6a837028 built 2019-07-24 14:33:55 +0000 UTC]
#

memory usage gets a little out of the hand (upwards 1G between hourly restarts), so meanwhile I did following "workaround" :

# crontab -l | tail -1
@hourly /bin/systemctl restart metricbeat.service
#

Alex_Kristiansen · September 10, 2019, 10:15pm

Alexus,

Can you paste us the config you're using? Has this started with 6.8.2?
Do you have a way of testing a newer version of Metricbeat to see if you run into the same memory issues?

alexus · September 11, 2019, 3:54am

I use "Beats central management", my config:

metricbeat.yml:

# grep -v ^# /etc/metricbeat/metricbeat.yml




management:
  enabled: true
  period: 1m0s
  events_reporter:
    period: 30s
    max_batch_size: 1000
  access_token: ${management.accesstoken}
  kibana:
    protocol: https
    host: x.x.x:443
    username: x
    password: x
    ssl: null
    timeout: 10s
    ignoreversion: true
  blacklist:
    output: console|file










#

I just upgraded my beats to 6.8.3:

# metricbeat version
metricbeat version 6.8.3 (amd64), libbeat 6.8.3 [9be0dc0ce65850ca0efb7310a87affa193a513a2 built 2019-08-29 18:13:26 +0000 UTC]
#

Alex_Kristiansen · September 11, 2019, 3:32pm

If you're using CM, what modules and metricsets do you have enabled? Are you still seeing the issue with 6.8.3?

alexus · September 11, 2019, 4:07pm

modules:

docker
rabbitmq
redis
system

in about hour since last time I restarted metricbeat.service:

systemctl status metricbeat.service output:

# systemctl status metricbeat.service 
● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
   Loaded: loaded (/lib/systemd/system/metricbeat.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-09-11 15:00:15 UTC; 1h 5min ago
     Docs: https://www.elastic.co/products/beats/metricbeat
 Main PID: 8907 (metricbeat)
    Tasks: 30 (limit: 4915)
   Memory: 520.4M
      CPU: 37min 29.201s
   CGroup: /system.slice/metricbeat.service
           └─8907 /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -path.home /usr/

Sep 11 15:00:15 app11 systemd[1]: Started Metricbeat is a lightweight shipper for metrics..
# metricbeat version
metricbeat version 6.8.3 (amd64), libbeat 6.8.3 [9be0dc0ce65850ca0efb7310a87affa193a513a2 built 2019-08-29 18:13:26 +0000 UTC]
#

Alex_Kristiansen · September 11, 2019, 5:44pm

@alexus

Sorry I keep asking for info, it's a bit hard to debug from CM. What output are you using? Are you using the add_kubernetes_metadata processor?

Also, can you get a memory profile? You can get one by adding -httpprof localhost:6060 to metricbeat, and then downloading http://localhost:6060/debug/pprof/heap. Wait until you start seeing high memory use, and then download it.

alexus · September 11, 2019, 8:00pm

Please... There is definitely no need to be sorry) I would love to help whichever way I can (help me, help you to help me)

I use elasticsearch for output and no I'm not using add_kubernetes_metadata (at least for now), how can I transfer heap file over to you?

Alex_Kristiansen · September 11, 2019, 8:25pm

@alexus,

It's been a while since I used the memory profiler. If it returns an image, you can just use the image attachment here. If not, maybe you can try a github gist, public s3 bucket or something like that?

system · October 9, 2019, 8:25pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unusually high Metricbeat memory usage Beats metricbeat	14	5289	August 7, 2019
Metricbeat extreme memory usage Beats metricbeat	2	330	April 28, 2021
Metricbeat using more than expected memory Beats metricbeat	7	591	December 29, 2020
MetricBeat v7.3.1 memory leak Beats metricbeat	9	1317	December 10, 2019
CPU usage increases over time meticbeat on Windows 10 Beats metricbeat	4	930	December 6, 2017

Memory usage is at extreme high level

Related topics