Filebeat Registry Issue handling large number of files

Elastic Stack Beats
1

Currently running filebeat 7.17.2.

I have a setup where filebeat is "scrapping" a folder where per minute 5 (or more in future) CSV files are dropped and filebeat exports to kafka. (CSV are TWAMP telemetry data)
Typically 5 days of files are kept to allow for backlog/ reprocessing should errors occur in NorthBound system consuming the information.

Doing some quick match for reference:

1H 300 files,
12H 3600 files,
24H 7200 files,
5D 36000 files.

Every time I need to restart filebeat I need to delete its registry because it simply will not do nothing for hours except going through the registry. Last time waited +4 hours and it never got to process incoming files even when deleted all files older than 1 day.

My current config:

# filebeat export config
filebeat:
  inputs:
  - clean_inactive: 24h
    clean_removed: true
    close_eof: true
    close_removed: true
    exclude_lines:
    - ^CSVexport Version
    harvester_limit: 10
    ignore_older: 12h
    paths:
    - /mnt/spool/kafka/*.csv
    type: log

Looking at metrics from 15 minute interval:

# grep metrics filebeat.log| tail -10 | cut -c98- | jq '.monitoring.metrics.registrar.states.current'
13389
13464
13538
13614
13689
13763
13838
13914
13989
14060

What I am doing wrong ?

From my config should the active count not stop at 7200 at the most ?

Thank you,

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.