New index not catching up with all data in the source files


(Sudhakar Shanmugam) #1

I face two different issue with index not being updated. I'm posting related filebeat and LS configs for those indexes, logs are barely helpful (even after turning them on DEBUG mode) to understand where the break is, whether shipping at Filebeat is broken or LS is broken or ES.

Filebeat.yml
-
paths:
- /opt/shared/reports/TransferSummary.*
scan_frequency: 3600s
fields:
log_type: transfersum_log
input_type: log
-
paths:
- /opt/shared/reports/DailyTransfersSummary.*
scan_frequency: 3600s
fields:
log_type: dailysum_log
input_type: log

I could see entries for all qualified reports under above location in registry, so I assume filebeat shipping the logs, is there any other place to check more?

logstash.conf have updated with grok filters and have been tested, it works. Also, its evident that it picked up partial data and indexed.

Issue#1: TransferSummary.* file updates every hour, however the index is not catching up with latest updates

Issue#2: DailyTransfersSummary.* files updates everyday, however the index pickedup the latest entry, however there are two files, one for this year and other for last year, not indexing all the data in these two files.

Please assist what am missing with these configurations.


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.