Currently running filebeat 7.17.2.
I have a setup where filebeat is "scrapping" a folder where per minute 5 (or more in future) CSV files are dropped and filebeat exports to kafka. (CSV are TWAMP telemetry data)
Typically 5 days of files are kept to allow for backlog/ reprocessing should errors occur in NorthBound system consuming the information.
Doing some quick match for reference:
1H 300 files,
12H 3600 files,
24H 7200 files,
5D 36000 files.
Every time I need to restart filebeat I need to delete its registry because it simply will not do nothing for hours except going through the registry. Last time waited +4 hours and it never got to process incoming files even when deleted all files older than 1 day.
My current config:
# filebeat export config filebeat: inputs: - clean_inactive: 24h clean_removed: true close_eof: true close_removed: true exclude_lines: - ^CSVexport Version harvester_limit: 10 ignore_older: 12h paths: - /mnt/spool/kafka/*.csv type: log
Looking at metrics from 15 minute interval:
# grep metrics filebeat.log| tail -10 | cut -c98- | jq '.monitoring.metrics.registrar.states.current' 13389 13464 13538 13614 13689 13763 13838 13914 13989 14060
What I am doing wrong ?
From my config should the active count not stop at 7200 at the most ?