# New index not catching up with all data in the source files

**URL:** <https://discuss.elastic.co/t/new-index-not-catching-up-with-all-data-in-the-source-files/71309>\
**Category:** Elasticsearch\
**Created:** [January 12, 2017, 4:19am UTC](https://discuss.elastic.co/t/new-index-not-catching-up-with-all-data-in-the-source-files/71309 "2017-01-12T04:19:17Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![shansudhakar](https://avatars.discourse-cdn.com/v4/letter/s/a9a28c/32.png) [@shansudhakar](https://discuss.elastic.co/u/shansudhakar)\
**Post date:** [January 12, 2017, 4:19am UTC](https://discuss.elastic.co/t/new-index-not-catching-up-with-all-data-in-the-source-files/71309/1 "2017-01-12T04:19:17Z")

</div>

I face two different issue with index not being updated. I'm posting related filebeat and LS configs for those indexes, logs are barely helpful (even after turning them on DEBUG mode) to understand where the break is, whether shipping at Filebeat is broken or LS is broken or ES.

Filebeat.yml  
-  
paths:  
- /opt/shared/reports/TransferSummary.\*  
scan\_frequency: 3600s  
fields:  
log\_type: transfersum\_log  
input\_type: log  
-  
paths:  
- /opt/shared/reports/DailyTransfersSummary.\*  
scan\_frequency: 3600s  
fields:  
log\_type: dailysum\_log  
input\_type: log

I could see entries for all qualified reports under above location in registry, so I assume filebeat shipping the logs, is there any other place to check more?

logstash.conf have updated with grok filters and have been tested, it works. Also, its evident that it picked up partial data and indexed.

Issue#1: TransferSummary.\* file updates every hour, however the index is not catching up with latest updates

Issue#2: DailyTransfersSummary.\* files updates everyday, however the index pickedup the latest entry, however there are two files, one for this year and other for last year, not indexing all the data in these two files.

Please assist what am missing with these configurations.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 9, 2017, 4:19am UTC](https://discuss.elastic.co/t/new-index-not-catching-up-with-all-data-in-the-source-files/71309/2 "2017-02-09T04:19:26Z")

</div>

Closed on February 9, 2017, 4:19am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.