Hello, I have an issue with memory and CPU consumption when starting filebeat. I have a log folder with over 100 files of around 100Mb each. I am not interest in all the files, as not all of them are current, so I have set ignore_older to 30m and harvester_limit to 1. However, when starting the service it reads the 109 files anyway and I'm struggling to understand why.
Here's my filebeat.yml fie
filebeat.inputs:
- type: log
enabled: true
paths:
- d:\Shared\log\*
#BRCSFPAEVSIP004
multiline.pattern: '^@.*message\ EventError.*'
multiline.negate: true
multiline.match: after
multiline.max_lines: 14
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
output.logstash:
hosts: ["localhost:5045"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
logging.level: info
ignore_older: 30m
scan_frequency: 5m
harvester_limit: 1
close_inactive: 10s
queue.mem:
events: 512
flush.min_events: 0
flush.timeout: 10s
Here's a sample of the filebeat log output:
2019-02-08T10:04:26.870-0200 INFO instance/beat.go:592 Home path: [D:\Downloads\Filebeat] Config path: [D:\Downloads\Filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\ProgramData\filebeat\logs]
2019-02-08T10:04:26.881-0200 INFO instance/beat.go:599 Beat UUID: 3f57613f-b37b-4c39-b3b0-029cb77ee882
2019-02-08T10:04:26.881-0200 INFO [beat] instance/beat.go:825 Beat info {"system_info": {"beat": {"path": {"config": "D:\\Downloads\\Filebeat", "data": "C:\\ProgramData\\filebeat", "home": "D:\\Downloads\\Filebeat", "logs": "C:\\ProgramData\\filebeat\\logs"}, "type": "filebeat", "uuid": "3f57613f-b37b-4c39-b3b0-029cb77ee882"}}}
2019-02-08T10:04:26.881-0200 INFO [beat] instance/beat.go:834 Build info {"system_info": {"build": {"commit": "bd8922f1c7e93d12b07e0b3f7d349e17107f7826", "libbeat": "6.5.4", "time": "2018-12-17T20:21:46.000Z", "version": "6.5.4"}}}
2019-02-08T10:04:26.882-0200 INFO [beat] instance/beat.go:837 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.10.6"}}}
2019-02-08T10:04:26.901-0200 INFO [beat] instance/beat.go:841 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-02-04T09:54:07.05-02:00","name":"DESKTOP-E6V977K","ip":["fe80::e459:5e64:88d8:85c/64","179.19.104.117/20","fe80::28bc:9b86:412f:1658/64","169.254.22.88/16","fe80::98c9:5ca3:1734:12f7/64","169.254.18.247/16","fe80::f928:f6d:3801:398f/64","169.254.57.143/16","fe80::4031:caf3:657f:b257/64","169.254.178.87/16","192.168.240.205/22","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17134.523 (WinBuild.160101.0800)","mac":["54:0b:aa:40:fb:12","54:bf:64:f5:f7:33","76:40:bb:74:34:bd","86:40:bb:74:34:bd","00:ff:6d:74:2f:6d","74:40:bb:74:34:bd"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Home Single Language","version":"10.0","major":10,"minor":0,"patch":0,"build":"17134.523"},"timezone":"-02","timezone_offset_sec":-7200,"id":"1c85de5e-d0e3-4063-b14f-d9a70f788b71"}}}
2019-02-08T10:04:26.903-0200 INFO [beat] instance/beat.go:870 Process info {"system_info": {"process": {"cwd": "C:\\Windows\\system32", "exe": "D:\\Downloads\\Filebeat\\filebeat.exe", "name": "filebeat.exe", "pid": 46892, "ppid": 892, "start_time": "2019-02-08T10:04:26.648-0200"}}}
2019-02-08T10:04:26.903-0200 INFO instance/beat.go:278 Setup Beat: filebeat; Version: 6.5.4
2019-02-08T10:04:29.921-0200 INFO add_cloud_metadata/add_cloud_metadata.go:319 add_cloud_metadata: hosting provider type not detected.
2019-02-08T10:04:29.922-0200 INFO [publisher] pipeline/module.go:110 Beat name: DESKTOP-E6V977K
2019-02-08T10:04:29.927-0200 INFO instance/beat.go:400 filebeat start running.
2019-02-08T10:04:29.927-0200 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2019-02-08T10:04:29.928-0200 INFO registrar/registrar.go:97 No registry file found under: C:\ProgramData\filebeat\registry. Creating a new registry file.
2019-02-08T10:04:29.940-0200 INFO registrar/registrar.go:134 Loading registrar data from C:\ProgramData\filebeat\registry
2019-02-08T10:04:29.940-0200 INFO registrar/registrar.go:141 States Loaded from registrar: 0
2019-02-08T10:04:29.940-0200 WARN beater/filebeat.go:374 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-02-08T10:04:29.941-0200 INFO crawler/crawler.go:72 Loading Inputs: 1
2019-02-08T10:04:29.941-0200 INFO log/input.go:138 Configured paths: [d:\Shared\log\*]
2019-02-08T10:04:29.941-0200 INFO input/input.go:114 Starting input of type: log; ID: 11326979680878536179
2019-02-08T10:04:29.941-0200 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2019-02-08T10:04:29.942-0200 INFO cfgfile/reload.go:150 Config reloader started
2019-02-08T10:04:29.942-0200 INFO cfgfile/reload.go:205 Loading of config files completed.
2019-02-08T10:05:00.187-0200 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":5562,"time":{"ms":5562}},"total":{"ticks":57640,"time":{"ms":57640},"value":0},"user":{"ticks":52078,"time":{"ms":52078}}},"handles":{"open":598},"info":{"ephemeral_id":"0f1ae7da-abef-4d8d-808b-b44127f39580","uptime":{"ms":33421}},"memstats":{"gc_next":439764672,"memory_alloc":353046480,"memory_total":13711576128,"rss":5108199424}},"filebeat":{"events":{"active":731,"added":750,"done":19},"harvester":{"open_files":109,"running":109,"started":109}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"active":3,"batches":3,"total":3},"read":{"bytes":30},"type":"logstash","write":{"bytes":2459}},"pipeline":{"clients":1,"events":{"active":533,"filtered":109,"published":532,"retry":1,"total":642}}},"registrar":{"states":{"current":19,"update":19},"writes":{"success":20,"total":20}},"system":{"cpu":{"cores":8}}}}}
Before starting the service, I deleted C:\ProgramData\filebeat to make sure that a new folder with a new registry would be created. I cleaned the harvester lines from the log as there were too many, but 109 harvesters were started as can be seen in the number of opened files in the last log line.
What kind of tweking should be made so that the number of harvesters open is actually 1?