# New data getting index with old data - Filebeat - Elasticsearch

**URL:** <https://discuss.elastic.co/t/new-data-getting-index-with-old-data-filebeat-elasticsearch/139444>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 11, 2018, 2:35am UTC](https://discuss.elastic.co/t/new-data-getting-index-with-old-data-filebeat-elasticsearch/139444 "2018-07-11T02:35:18Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![nandan\_bigdata](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@nandan\_bigdata](https://discuss.elastic.co/u/nandan_bigdata)\
**Post date:** [July 11, 2018, 2:35am UTC](https://discuss.elastic.co/t/new-data-getting-index-with-old-data-filebeat-elasticsearch/139444/1 "2018-07-11T02:35:18Z")

</div>

Hello Everyone,  
In my architecture, I am using 3 elastic components such as:-

1. Filebeat
2. Elasticsearch
3. Kibana

filebeat.yml configuration file is :-

> #=========================== Filebeat inputs =============================  
> filebeat.inputs:
> 
> - type: log  
> enabled: true  
> paths:
> - /var/log/nandan.log  
> #============================= Filebeat modules ===============================  
> filebeat.config.modules:  
> path: ${path.config}/modules.d/\*.yml  
> reload.enabled: true  
> #==================== Elasticsearch template setting ==========================  
> setup.template.settings:  
> index.number\_of\_shards: 3
> 
> #-------------------------- Elasticsearch output ------------------------------  
> output.elasticsearch:  
> hosts: ["172.16.101.107:9200"]

and my input file "nandan.log" data getting an index into Elasticsearch successfully. But the problem is  
whenever I am inserting new data inside **"nandan.log"** file, Elasticsearch is getting an index of all new data as well as old data every time.  
**_Right now in "nandan.log" file, there are 100-150 lines but in elasticsearch index,, there are 750 hits showing._**  
Please tell me how to resolve this.

Thanks

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 11, 2018, 5:24am UTC](https://discuss.elastic.co/t/new-data-getting-index-with-old-data-filebeat-elasticsearch/139444/2 "2018-07-11T05:24:01Z")

</div>

Exactly how are you inserting new data to the file? If you are using an editor it is quite likely this will create a new file and then replace the old one. This will appear as a new file to Filebeat, so it will be reprocessed from the beginning. Make sure that you are appending data to the existing file, e.g. `echo "Log entry" >> /var/log/nandan.log`.

---

<div class="post-metadata">

**Author:** ![nandan\_bigdata](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@nandan\_bigdata](https://discuss.elastic.co/u/nandan_bigdata)\
**Post date:** [July 11, 2018, 6:07am UTC](https://discuss.elastic.co/t/new-data-getting-index-with-old-data-filebeat-elasticsearch/139444/3 "2018-07-11T06:07:34Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> tly how are you inserting new data to the file? If you are using an editor it is quite likely this will create a new file and then replace the old one. This will appear as a new file to Filebeat, so it will be reprocessed from the beginning. Make s

Thanks ,\> Got your solution.  
I was using vi editor fro inserting new data.  
after that I tried to use echo method and it is working well.

I am just trying to find solution for real time log analysis.

Thanks

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 11, 2018, 6:10am UTC](https://discuss.elastic.co/t/new-data-getting-index-with-old-data-filebeat-elasticsearch/139444/4 "2018-07-11T06:10:30Z")

</div>

Logs written to by applications are typically appended to by default.

---

<div class="post-metadata">

**Author:** ![nandan\_bigdata](https://avatars.discourse-cdn.com/v4/letter/n/f07891/32.png) [@nandan\_bigdata](https://discuss.elastic.co/u/nandan_bigdata)\
**Post date:** [July 11, 2018, 6:15am UTC](https://discuss.elastic.co/t/new-data-getting-index-with-old-data-filebeat-elasticsearch/139444/5 "2018-07-11T06:15:29Z")

</div>

Yes correct.. So Filebeat will get data and new log data will go index automatically..  
For this I checked as echo command..

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 8, 2018, 6:15am UTC](https://discuss.elastic.co/t/new-data-getting-index-with-old-data-filebeat-elasticsearch/139444/6 "2018-08-08T06:15:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.