Hey,
Running a filebeat on a NFS share which stores backup of json message files from our application's communication with external applications and wonder how to best avoid registery to blow up in size while also ensuring no data loss.
Message files are only written once and organized into daily directories, and maybe count +100k per day in production.
Not quite sure what best to configure of ignore_*, close.* and clean_* options at.
Not happy about clean_removed: true vs our soft mount NFS share, should we go for hard mounted and/or clean_removed: false + other ignore_* + close.* options both to avoid reingesting and data loss and keeping registery from blowning up in size. Hints appreciated, TIA.
Below current config from a test bed across multiple test environment all written to the same NFS share and all harvest by a single filebeat instance:
[root@exrhel0311 ~]$ ps -e -o vsz,pid,args -q 1243,1245
VSZ PID COMMAND
20569828 1243 /usr/share/pjp/filebeat/bin/pjp_filebeat --environment systemd -c /etc/pjp/filebeat/filebeat.yml --path.home /usr/share/pjp/filebeat --path.config /etc/pjp/fil
2038152 1245 /usr/share/pjp/metricbeat/bin/pjp_metricbeat -c /etc/pjp/metricbeat/metricbeat.yml -path.home /usr/share/pjp/metricbeat -path.config /etc/pjp/metricbeat -path.d
root@exrhel0311 ~]$ wc -l /var/lib/pjp/filebeat/data/registry/filebeat/26426764.json
616002 /var/lib/pjp/filebeat/data/registry/filebeat/26426764.json
[root@exrhel0311 ~]$ cat /etc/pjp/filebeat/filebeat.yml
# ============================== Filebeat inputs ===============================
filebeat.inputs:
### CAMEL Backup messages ###
- type: filestream
id: camel-message-backup
enabled: true
pipeline: pjp-camel-message-backup
fields.index: pjp_camel_backup
close.reader.on_eof: true
clean_removed: true
harvester_limit: 400
file_identity.path: ~
paths:
# Only look ahead of us the next 7 years, load manually if the past (pre oct 2023) are of interest
# revise this no later than in 2029 ;)
- /camel_backup/*/20231*/*/*/*.json
- /camel_backup/*/202[4-9]*/*/*/*.json
tags:
- camel
- message-backup
parsers:
- multiline:
type: pattern
pattern: '^{'
negate: true
match: after
max_lines: 5000
processors:
- script:
lang: javascript
source: >
function process(event) {
var message = event.Get('message')
if (message.charAt(message.length-1) != '}') {
var message = message + '}'
}
event.Put('message', message)
}