Filebeat large number of files opened

Elastic Stack Beats
1

Have issue on 1 machine sending logs from filebeat to kafka, it's lagging a lot and restarting a filebeat takes more than 45 minutes to completely restart it.

- clean_inactive: 18h
  close_removed: true
  close_inactive: 5m
  ignore_older: 12h
  type: log
  enabled: true
  fields_under_root: true
  fields:
    kafka_topic: logs-1
  paths:
  - "/path/logs/*/main.log.????-??-??-??"
  - "path/logs/*/main.log"
  - "path/logs/*/main.log.????-??-??"
  file_identity.inode_marker.path: "/path/logs/.filebeat-marker"
  multiline.type: pattern
  multiline.negate: true
  multiline.match: after
  multiline.pattern: "^\\d\\d\\d\\d-\\d\\d-\\d\\d\\s+\\d\\d:\\d\\d:\\d\\d"

output.kafka:
  hosts:
  - kafka1:9092
  ...
  - kafka32:9092
  loadbalance: true
  topic: "%{[kafka_topic]}"
  compression: snappy
  max_message_bytes: 1024000
  ssl.enabled: true
  sasl.mechanism: PLAIN
  workers: 1

logs are being rotated every 1 hour, kafka has 32 brokers, i see a lag of 4 hours, and number of files opened by filebeat is over 90k. According to ignore_older and clean_inactive, fb should ignore everything that is older than 18 hours, but for some reason even files from 7 days ago are opened, it consumes lot of cpu and throughput is not well. Any tips what should I check, and maybe tune some parameteres like number of workers or? The number of files changed in the last hour with that name patter is around 6000.

On a side note, I have really large number of same machines doing the same thing, configuration is the same on all of them, but only this one has issues because it has most files on it.

2

Hi @ansamHox

I reviewed your flag

turned out not to be filebeat error at all, but logstash wasn't processing, it's irrelevant, thnx

That's information that could actually be helpful to other users, so we would prefer to leave the post.

Thanks!

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.