Filebeat CPU usage too high

Elastic Stack Beats
1

Hi,

My filebeats' CPU usage is too high.
I couldn't find solution in this forum.
I'm suspecting multiple wildcard inclusion such as *-*_*.log causes CPU consumption.

Please tell me how to reduce CPU usage.
Version of elk stack is 7.2.1

Part of filebeat.yml(main config is in the foo.yml)

filebeat.inputs:
- type: log
  enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1

#========================= Filebeat global options ============================

# Enable filebeat config reloading
filebeat.config:
  inputs:
    enabled: true
    path: configs/*.yml
    reload.enabled: true
    reload.period: 10s

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["foobar:5044"]

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

#============================== Xpack Monitoring ===============================
#monitoring.enabled: false
xpack.monitoring.enabled: true

#monitoring.elasticsearch:
xpack.monitoring.elasticsearch:
  #hosts: ["localhost:9200"]
  hosts: ["foobar:9200"]
  metrics.period: 15m
  state.period: 15m

#================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

Part of configs/foo.yml

- type: log
  enabled: true
  paths:
    - D:\Program Files (x86)\dummy\dummy\dummy\dummy*
  encoding: Shift_JIS
  fields: {tags: dummy-common}

- type: log
  enabled: true
  paths:
    - D:\dummy dummy\dummy\dummy\dummy\dummy*.csv
  encoding: Shift_JIS
  fields: {tags: dummy-dummy}

- type: log
  enabled: true
  paths:
    - /dummy/dummy/dummy/dummy/dummy.csv
  encoding: utf-8
  fields: {tags: dummy-dummy}

- type: log
  enabled: true
  paths:
    - F:\dummy\dummy\dummy\dummy\dummy*.LOG
  encoding: Shift_JIS
  fields: {tags: dummy-dummy-dummy}
  multiline:
    pattern: '^[0-9]{4}\/[0-9]{2}\/[0-9]{2}|^[0-9]{2}\/[0-9]{2}\/[0-9]{2}'
    negate: true
    match: after

- type: log
  enabled: true
  paths:
    - /var/dummy/dummy/dummy.LOG
  encoding: utf-8
  fields: {tags: dummy-dummy-dummy}
  multiline:
    pattern: '^[0-9]{4}\/[0-9]{2}\/[0-9]{2}|^[0-9]{2}\/[0-9]{2}\/[0-9]{2}'
    negate: true
    match: after


- type: log
  enabled: true
  paths:
    - F:\dummy\dummy\dummy\dummy\dummy*.*.dummy
  encoding: Shift_JIS
  fields: {tags: dummy-dummy}
  multiline:
    pattern: '^\|\s'
    negate: false
    match: after
  include_lines: ['^\+']
  exclude_lines: ['\+$']

- type: log
  enabled: true
  paths:
    - F:\dummy\dummy\dummy\dummy\dummy*.*.dummy
  encoding: Shift_JIS
  fields: {tags: dummy-dummy}
  multiline:
    pattern: '^dummy|^\+|^\|'
    negate: true
    match: after
    max_lines : 2000
  exclude_lines: ['^\+|^\|']

- type: log
  enabled: true
  paths:
    - /dummy/dummy/dummy/*.dummy
  encoding: utf-8
  fields: {tags: dummy-dummy}
  multiline:
    pattern: '^\|\s'
    negate: false
    match: after
  include_lines: ['^\+']
  exclude_lines: ['\+$']

- type: log
  enabled: true
  paths:
    - G:\dummy\dummy\dummy\dummy.log*
  encoding: Shift_JIS
  fields: {tags: dummy-queue}

- type: log
  enabled: true
  paths:
    - /dummy/dummy/dummy/dummy/dummy.log
  encoding: utf-8
  fields: {tags: dummy-queue}

- type: log
  enabled: true
  paths:
    - G:\dummy\dummy\dummy\dummy\dummy\dummy_*_*-*.log
  encoding: Shift_JIS
  fields: {tags: dummy-err}

- type: log
  enabled: true
  paths:
    - G:\dummy\dummy\dummy\dummy\dummy\dummy_*_*-*.log
  encoding: Shift_JIS
  fields: {tags: dummy-workflow}
  multiline:
    pattern: '^.*?:.*?[[:space:]]\['
    negate: true
    match: after
2

Each logs are generated max 1log/day, and less than 5mb.
It seems CPU usage keeps glowing after log shipping completion.

3

I found cause of this probrem.
My filebeat conig has the same entries in one file like below.
Once I deleted one of these entries, cpu usage was stable at low.

- type: log
  enabled: true
  paths:
    - G:\dir1\dir2\dir2\dir3\dir4\s_*_*-*.log
  encoding: Shift_JIS
  fields: {tags: tag-session}
  multiline:
    pattern: '^\*\>[[:space:]]'
    negate: false
    match: after

- type: log
  enabled: true
  paths:
    - G:\dir1\dir2\dir2\dir3\dir4\s_*_*-*.log
  encoding: Shift_JIS
  fields: {tags: tag-session}
  multiline:
    pattern: '^\*\>[[:space:]]'
    negate: false
    match: after

I can close this issue, but I want to know this misconfigure at filebeat's boot stage.
Could anyone tell me how I can know this misconfigure?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.