Hello,
I tried to tune filebeat performance, as @steffens suggested in one post, i changed the output to console, one thing I find strange is: I allocated 3 cores to filebeat, however, i can only drive the CPU usage to two cores at 50% and one core at 20%. we are using filebeat 6.1.3.
why i can not drive CPU usage close to 100%? what might be the bottleneck? any tips in debugging the bottleneck would be appreciated.
log stats is:
2019-07-10T18:11:08-07:00 INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30038 beat.memstats.gc_next=59367392 beat.memstats.memory_alloc=54714200 beat.memstats.memory_total=12811647016 filebeat.events.active=1247 filebeat.events.added=180043 filebeat.events.done=178796 filebeat.harvester.open_files=3 filebeat.harvester.running=3 filebeat.harvester.started=3 libbeat.config.module.running=0 libbeat.output.events.acked=178790 libbeat.output.events.batches=88 libbeat.output.events.total=178790 libbeat.output.type=console libbeat.output.write.bytes=74209763 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=1247 libbeat.pipeline.events.filtered=6 libbeat.pipeline.events.published=180037 libbeat.pipeline.events.total=180043 libbeat.pipeline.queue.acked=178790 registrar.states.current=6 registrar.states.update=178796 registrar.writes=94
2019-07-10T18:11:38-07:00 INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=29985 beat.memstats.gc_next=43650096 beat.memstats.memory_alloc=45868008 beat.memstats.memory_total=14817652576 filebeat.events.active=-1247 filebeat.events.added=174213 filebeat.events.done=175460 filebeat.harvester.open_files=3 filebeat.harvester.running=3 libbeat.config.module.running=0 libbeat.output.events.acked=175460 libbeat.output.events.batches=86 libbeat.output.events.total=175460 libbeat.output.write.bytes=73454908 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=0 libbeat.pipeline.events.published=174213 libbeat.pipeline.events.total=174213 libbeat.pipeline.queue.acked=175460 registrar.states.current=6 registrar.states.update=175460 registrar.writes=86
so events added is roughly the same as events done, so why filebeat could not add events fast enough?
filebeat yaml file:
#=========================== Filebeat prospectors =============================
filebeat.prospectors:
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
- input_type: log
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /export/content/lid/apps/watchmen-server/i001/logs/*
# closes the harvester/fd when a file is removed.
close_removed: true
# filebeat will focus on sending the current log files
ignore_older: 60m
close_inactive: 5m
# set max lifetime of each harvester (fd)
close_timeout: 70m
#after clean, the file content will be sent again from the beginning if the same file name re-appear
#for sparse logs, we do not mind dup in the kafka, kafka will discard old logs > 5 days
clean_inactive: 48h
clean_removed: true
symlinks: true
### Multiline options
# Mutiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
multiline.pattern: '^[[:digit:]]{4,}[[:punct:]][[:digit:]]{2,}|^[I|W|E|F|D|T][[:digit:]]{4,}'
# Defines if the pattern set under pattern should be negated or not. Default is false.
multiline.negate: true
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
multiline.match: after
#================================ processor =====================================
processors:
- drop_fields:
fields: ["beat.name", "beat.version", "input_type", "@timestamp", "@metadata", "metadata.type", "type", "offset"]
#-------------------------- Kafka output ------------------------------
output.console:
pretty: false
But this is just some guessing work I can't confirm from 'remote debugging'.