Hi Everyone,
I'm new to Elasticsearch and have been facing an issue with filebeats where I've been observing that logs from certain paths do not seem to get picked when the file in that path is updated. Running filebeat in debugger mode also doesn't show up anything relevant to that path.
However, upon deleting the registry file, filebeat detects the file. This I can say as I see a log message that the harvester has started on the file.
A snippet of how the YAML configuration for the concerned path appear is below.
- type: log
enabled: true
paths:
- /data/hddata*/hadoop-yarn/container/application_*/container*/*
fields:
log_type: log
ignore_older: 2h
scan_frequency: 1s
close_inactive: 1m
close_timeout: 5m
fields_under_root: true