Hello,
On a daily basis, we are exporting 4 configurations files. We are overwriting the same file with the new data that is pretty much the same every time.
My problem is that filebeat does not realize it is a new file and I'd like it to send the whole file on a daily basis.
Some times, it sends all the data, some times just a subset of it. Right now, I'm deleting the existing file to recreate one (happening in few seconds). I also try to erase the old file and create a new file with a new name (filebeat sees it as a file being renamed).
My latest configuration file looks like this:
- type: filestream
id: ConfigurationFile_1
paths:
- /var/config/ConfigurationFile_1.csv
clean_removed: true
prospector:
scanner:
resend_on_touch: true
fingerprint:
enabled: true
offset: 0
length: 1024
# file_identity:
# fingerprint:
# enabled: true
# offset: 0
# length: 1024
ignore_older: 3m
fields:
log_topic: configuration_kafka_topic
processors:
- add_fields:
target: ""
fields:
system_name: as002
- dissect:
tokenizer: "%{enterpriseId}|%{enterpriseName}|%{groupId}|%{groupName}|%{domainName}|"
field: "message"
target_prefix: "SystemConfig"
trim_values: "all"
- script:
lang: javascript
id: lowercase
source: >
function process(event) {
var groupId = event.Get("SystemConfig.groupId");
if (groupId != null) {
event.Put("SystemConfig.groupIdLowerCase", groupId.toLowerCase());
}
}