Registery blow ups


Running a filebeat on a NFS share which stores backup of json message files from our application's communication with external applications and wonder how to best avoid registery to blow up in size while also ensuring no data loss.

Message files are only written once and organized into daily directories, and maybe count +100k per day in production.

Not quite sure what best to configure of ignore_*, close.* and clean_* options at.
Not happy about clean_removed: true vs our soft mount NFS share, should we go for hard mounted and/or clean_removed: false + other ignore_* + close.* options both to avoid reingesting and data loss and keeping registery from blowning up in size. Hints appreciated, TIA.

Below current config from a test bed across multiple test environment all written to the same NFS share and all harvest by a single filebeat instance:

[root@exrhel0311 ~]$ ps -e -o vsz,pid,args -q 1243,1245
20569828 1243 /usr/share/pjp/filebeat/bin/pjp_filebeat --environment systemd -c /etc/pjp/filebeat/filebeat.yml --path.home /usr/share/pjp/filebeat --path.config /etc/pjp/fil
2038152 1245 /usr/share/pjp/metricbeat/bin/pjp_metricbeat -c /etc/pjp/metricbeat/metricbeat.yml -path.home /usr/share/pjp/metricbeat -path.config /etc/pjp/metricbeat -path.d

root@exrhel0311 ~]$ wc -l /var/lib/pjp/filebeat/data/registry/filebeat/26426764.json
616002 /var/lib/pjp/filebeat/data/registry/filebeat/26426764.json

[root@exrhel0311 ~]$ cat /etc/pjp/filebeat/filebeat.yml
# ============================== Filebeat inputs ===============================


### CAMEL Backup messages ###

- type: filestream
  id: camel-message-backup
  enabled: true
  pipeline: pjp-camel-message-backup
  fields.index: pjp_camel_backup
  close.reader.on_eof: true
  clean_removed: true
  harvester_limit: 400
  file_identity.path: ~
    # Only look ahead of us the next 7 years, load manually if the past (pre oct 2023) are of interest
    # revise this no later than in 2029 ;)
    - /camel_backup/*/20231*/*/*/*.json
    - /camel_backup/*/202[4-9]*/*/*/*.json
    - camel
    - message-backup
    - multiline:
        type: pattern
        pattern: '^{'
        negate: true
        match: after
        max_lines: 5000
    - script:
        lang: javascript
        source: >
          function process(event) {
            var message = event.Get('message')
            if (message.charAt(message.length-1) != '}') {
              var message = message + '}'
            event.Put('message', message)

I think this is the main issue, this is not recommend and can lead to many unexpected issues, which could be your case.

Ignore the NFS share and let's focus on handling the +100K/day written-once files vs minimizing the registery memory size, any hints on closing + removing such best?

i'm not sure you can just ignore the fact that you are using a NFS share since Filebeat does not work well with NFS share and have some issues with it.

All the configurations you could set expect that you are not using a network share, so they may not work well on a network share.

Since your files are written only once, you are already using the recommended setting to close on EOF.

Maybe changing the file_identity to fingerprint can help in this case, but I'm not sure as I do not use Filebeat with network shares, so you would need to test it yourself.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.