Filebeat High Memory and CPU usage


(Raul Ferreira De Souza) #1

Hello, I have an issue with memory and CPU consumption when starting filebeat. I have a log folder with over 100 files of around 100Mb each. I am not interest in all the files, as not all of them are current, so I have set ignore_older to 30m and harvester_limit to 1. However, when starting the service it reads the 109 files anyway and I'm struggling to understand why.

Here's my filebeat.yml fie

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - d:\Shared\log\*
    #BRCSFPAEVSIP004
  multiline.pattern: '^@.*message\ EventError.*'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 14
  
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.logstash:
  hosts: ["localhost:5045"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

logging.level: info
ignore_older: 30m
scan_frequency: 5m
harvester_limit: 1
close_inactive: 10s
queue.mem:
  events: 512
  flush.min_events: 0
  flush.timeout: 10s

Here's a sample of the filebeat log output:

2019-02-08T10:04:26.870-0200	INFO	instance/beat.go:592	Home path: [D:\Downloads\Filebeat] Config path: [D:\Downloads\Filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\ProgramData\filebeat\logs]
2019-02-08T10:04:26.881-0200	INFO	instance/beat.go:599	Beat UUID: 3f57613f-b37b-4c39-b3b0-029cb77ee882
2019-02-08T10:04:26.881-0200	INFO	[beat]	instance/beat.go:825	Beat info	{"system_info": {"beat": {"path": {"config": "D:\\Downloads\\Filebeat", "data": "C:\\ProgramData\\filebeat", "home": "D:\\Downloads\\Filebeat", "logs": "C:\\ProgramData\\filebeat\\logs"}, "type": "filebeat", "uuid": "3f57613f-b37b-4c39-b3b0-029cb77ee882"}}}
2019-02-08T10:04:26.881-0200	INFO	[beat]	instance/beat.go:834	Build info	{"system_info": {"build": {"commit": "bd8922f1c7e93d12b07e0b3f7d349e17107f7826", "libbeat": "6.5.4", "time": "2018-12-17T20:21:46.000Z", "version": "6.5.4"}}}
2019-02-08T10:04:26.882-0200	INFO	[beat]	instance/beat.go:837	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.10.6"}}}
2019-02-08T10:04:26.901-0200	INFO	[beat]	instance/beat.go:841	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-02-04T09:54:07.05-02:00","name":"DESKTOP-E6V977K","ip":["fe80::e459:5e64:88d8:85c/64","179.19.104.117/20","fe80::28bc:9b86:412f:1658/64","169.254.22.88/16","fe80::98c9:5ca3:1734:12f7/64","169.254.18.247/16","fe80::f928:f6d:3801:398f/64","169.254.57.143/16","fe80::4031:caf3:657f:b257/64","169.254.178.87/16","192.168.240.205/22","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17134.523 (WinBuild.160101.0800)","mac":["54:0b:aa:40:fb:12","54:bf:64:f5:f7:33","76:40:bb:74:34:bd","86:40:bb:74:34:bd","00:ff:6d:74:2f:6d","74:40:bb:74:34:bd"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Home Single Language","version":"10.0","major":10,"minor":0,"patch":0,"build":"17134.523"},"timezone":"-02","timezone_offset_sec":-7200,"id":"1c85de5e-d0e3-4063-b14f-d9a70f788b71"}}}
2019-02-08T10:04:26.903-0200	INFO	[beat]	instance/beat.go:870	Process info	{"system_info": {"process": {"cwd": "C:\\Windows\\system32", "exe": "D:\\Downloads\\Filebeat\\filebeat.exe", "name": "filebeat.exe", "pid": 46892, "ppid": 892, "start_time": "2019-02-08T10:04:26.648-0200"}}}
2019-02-08T10:04:26.903-0200	INFO	instance/beat.go:278	Setup Beat: filebeat; Version: 6.5.4
2019-02-08T10:04:29.921-0200	INFO	add_cloud_metadata/add_cloud_metadata.go:319	add_cloud_metadata: hosting provider type not detected.
2019-02-08T10:04:29.922-0200	INFO	[publisher]	pipeline/module.go:110	Beat name: DESKTOP-E6V977K
2019-02-08T10:04:29.927-0200	INFO	instance/beat.go:400	filebeat start running.
2019-02-08T10:04:29.927-0200	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2019-02-08T10:04:29.928-0200	INFO	registrar/registrar.go:97	No registry file found under: C:\ProgramData\filebeat\registry. Creating a new registry file.
2019-02-08T10:04:29.940-0200	INFO	registrar/registrar.go:134	Loading registrar data from C:\ProgramData\filebeat\registry
2019-02-08T10:04:29.940-0200	INFO	registrar/registrar.go:141	States Loaded from registrar: 0
2019-02-08T10:04:29.940-0200	WARN	beater/filebeat.go:374	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-02-08T10:04:29.941-0200	INFO	crawler/crawler.go:72	Loading Inputs: 1
2019-02-08T10:04:29.941-0200	INFO	log/input.go:138	Configured paths: [d:\Shared\log\*]
2019-02-08T10:04:29.941-0200	INFO	input/input.go:114	Starting input of type: log; ID: 11326979680878536179 
2019-02-08T10:04:29.941-0200	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
2019-02-08T10:04:29.942-0200	INFO	cfgfile/reload.go:150	Config reloader started
2019-02-08T10:04:29.942-0200	INFO	cfgfile/reload.go:205	Loading of config files completed.
2019-02-08T10:05:00.187-0200	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":5562,"time":{"ms":5562}},"total":{"ticks":57640,"time":{"ms":57640},"value":0},"user":{"ticks":52078,"time":{"ms":52078}}},"handles":{"open":598},"info":{"ephemeral_id":"0f1ae7da-abef-4d8d-808b-b44127f39580","uptime":{"ms":33421}},"memstats":{"gc_next":439764672,"memory_alloc":353046480,"memory_total":13711576128,"rss":5108199424}},"filebeat":{"events":{"active":731,"added":750,"done":19},"harvester":{"open_files":109,"running":109,"started":109}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"active":3,"batches":3,"total":3},"read":{"bytes":30},"type":"logstash","write":{"bytes":2459}},"pipeline":{"clients":1,"events":{"active":533,"filtered":109,"published":532,"retry":1,"total":642}}},"registrar":{"states":{"current":19,"update":19},"writes":{"success":20,"total":20}},"system":{"cpu":{"cores":8}}}}}

Before starting the service, I deleted C:\ProgramData\filebeat to make sure that a new folder with a new registry would be created. I cleaned the harvester lines from the log as there were too many, but 109 harvesters were started as can be seen in the number of opened files in the last log line.

What kind of tweking should be made so that the number of harvesters open is actually 1?


(Raul Ferreira De Souza) #2

UPDATE:

The problem was solved by formatting the yml file, putting the limiting options under the multine options I already had:

filebeat.inputs:
- type: log
  enabled: true
  paths:
- d:\Shared\log\*
#BRCSFPAEVSIP004
  multiline.pattern: '^@.*message\ EventError.*'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 14
  ignore_older: 30m
  scan_frequency: 5m
  harvester_limit: 1
  close_inactive: 10s
  
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.logstash:
  hosts: ["localhost:5045"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~