2 days ago i did an upgrade from Logstash and Filebeat from 5.6.x to 6.0, then i got some fun time with index_templates and document_type, but after that, i found the strange behavior of Filebeat.
i only changed the document_type to tags
- type: log tags: ["report_daily"] paths: - /usr/src/the-report/report_daily*.csv - type: log tags: ["report_month"] paths: - /usr/src/the-report/report_month*.csv
then i was looking for this problem and added these:
- type: log tags: ["report_daily"] paths: - /usr/src/the-report/report_daily*.csv ignore_older: 1h close_inactive: 30s - type: log tags: ["report_month"] paths: - /usr/src/the-report/report_month*.csv ignore_older: 1h close_inactive: 30s
both configs didn't change it after Filebeat is done with the file (based on the amount in Elasticsearch/Kibana), it will start it again.
for example, the month index should be around 196k docs and now it will raise up to 380 or higher...
About the .csv file:
it will be every hour be moved away into an archive folder and if there were changes in the source, where i pull the .csv information, i will throw it with a new name into the log (report) folder, that Filebeat can grab it.
Sure i could rollback to 5.6.x, but would prefer to understand the issue
Ah, here is the registry
appreciate any help, thanks in advance