New data getting index with old data - Filebeat - Elasticsearch

[nandan_bigdata (ELK Explorer)]

Hello Everyone,
In my architecture, I am using 3 elastic components such as:-

1. Filebeat
2. Elasticsearch
3. Kibana

filebeat.yml configuration file is :-

#=========================== Filebeat inputs =============================
filebeat.inputs:

• type: log
  enabled: true
  paths:
  • /var/log/nandan.log
    #============================= Filebeat modules ===============================
    filebeat.config.modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: true
    #==================== Elasticsearch template setting ==========================
    setup.template.settings:
    index.number_of_shards: 3

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
hosts: ["172.16.101.107:9200"]

and my input file "nandan.log" data getting an index into Elasticsearch successfully. But the problem is
whenever I am inserting new data inside "nandan.log" file, Elasticsearch is getting an index of all new data as well as old data every time.
Right now in "nandan.log" file, there are 100-150 lines but in elasticsearch index,, there are 750 hits showing.
Please tell me how to resolve this.

Thanks

[Christian_Dahlqvist]

Exactly how are you inserting new data to the file? If you are using an editor it is quite likely this will create a new file and then replace the old one. This will appear as a new file to Filebeat, so it will be reprocessed from the beginning. Make sure that you are appending data to the existing file, e.g. echo "Log entry" >> /var/log/nandan.log.

[nandan_bigdata (ELK Explorer)]

@Christian_Dahlqvist

tly how are you inserting new data to the file? If you are using an editor it is quite likely this will create a new file and then replace the old one. This will appear as a new file to Filebeat, so it will be reprocessed from the beginning. Make s

Thanks ,> Got your solution.
I was using vi editor fro inserting new data.
after that I tried to use echo method and it is working well.

I am just trying to find solution for real time log analysis.

Thanks

[Christian_Dahlqvist]

Logs written to by applications are typically appended to by default.

[nandan_bigdata (ELK Explorer)]

Yes correct.. So Filebeat will get data and new log data will go index automatically..
For this I checked as echo command..