Filebeat high read io when collecting log data

Elastic Stack Beats
1

I have a Filebeat setup to collect log data in SATA, but it does have a high read IO of around 20mb/s. below is my Filebeat config, it outputs to Kafka cluster. im just wondering is there any way to slow down the Filebeat collecting rate to reduce the read IO on disk? or any config that I can make to lower the IO?

  type: log
  encoding: plain
  scan_frequency: 3s
  paths:
    - /usr/share/filebeat/logs/*/*.log
    - /usr/share/filebeat/logs/*/*.error
  fields_under_root: true
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  multiline.timeout: 5s
  tail_files: false
  symlinks: false
  backoff: 1s
  max_backoff: 10s
  backoff_factor: 2
  harvester_limit: 0
  ### Harvester closing options
  close_inactive: 20m
  close_renamed: false
  close_removed: false
  close_eof: false
  close_timeout: 0
  ### State options
  clean_inactive: 0
  clean_removed: true

output.kafka:
  enabled: true
  hosts: '${KAFKA_HOSTS:kafka:9092}'
  topic: 'topic'
  partition.round_robin:
    reachable_only: true
2

Check this and this

I would try to set values next params and test:

queue.mem:
  events: 4096
  flush.min_events: 512
  flush.timeout: 5s
4

sure, ill give it a try.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.