Registry log.json grows constantly even with filebeat.registry.flush: 60s

Elastic Stack Beats
1

Filebeat 7.17.1

I'm looking into decreasing the amount of IO for the filebeat registry in our deployment. I found the filebeat.registry.flush setting after some research and thought that it would do what I wanted. I set it to 60s redeployed and monitored the registry directory. I didn't see what I expected as I found that the log.json constantly grows at least per second if not faster. I was expecting to see a gap of 60 seconds between the file modifications. I'm guessing my understanding is incorrect though?

I also tried changing the setting to 5s as a test to see if it was a problem with the amount of time. No difference...

Here is my filebeat.yml

filebeat.config.inputs:
  enabled: true
  path: /etc/filebeat/inputs.yml
  reload.enabled: true
  reload.period: 10s

filebeat.registry.path: /var/resin/logs/${REALM_CMID}/registry
filebeat.registry.flush: 5s
#================================ Outputs =====================================
output.logstash:
  hosts: ["${FILEBEAT_LOGSTASH_HOST}:5044"]
  ssl.certificate_authorities: ["/etc/ssl/certs/${RESIN_SSL_CERTIFICATE_NAME}"]
  ssl.verification_mode: none

I know the changes that I made are being read as I also changed the registry path and that did take effect.

We have approx 30 files being monitored. The input config for each is almost identical:

- type: filestream
  id: pok-account
  enabled: true
  fields:
    log_type: account
    product: pok
    kingdom: ${KINGDOM_CMID}
    realm: ${REALM_CMID}
  parsers:
    - multiline:
        type: pattern
        pattern: '^\['
        negate: true
        match: after
  paths:
    - /var/resin/logs/*/pok/log/*account.log

Any assistance or clarification would be greatly appreciated.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.