New data getting index with old data - Filebeat - Elasticsearch

Elastic Stack Beats
1

Hello Everyone,
In my architecture, I am using 3 elastic components such as:-

  1. Filebeat
  2. Elasticsearch
  3. Kibana

filebeat.yml configuration file is :-

#=========================== Filebeat inputs =============================
filebeat.inputs:

  • type: log
    enabled: true
    paths:
    • /var/log/nandan.log
      #============================= Filebeat modules ===============================
      filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: true
      #==================== Elasticsearch template setting ==========================
      setup.template.settings:
      index.number_of_shards: 3

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
hosts: ["172.16.101.107:9200"]

and my input file "nandan.log" data getting an index into Elasticsearch successfully. But the problem is
whenever I am inserting new data inside "nandan.log" file, Elasticsearch is getting an index of all new data as well as old data every time.
Right now in "nandan.log" file, there are 100-150 lines but in elasticsearch index,, there are 750 hits showing.
Please tell me how to resolve this.

Thanks

2

Exactly how are you inserting new data to the file? If you are using an editor it is quite likely this will create a new file and then replace the old one. This will appear as a new file to Filebeat, so it will be reprocessed from the beginning. Make sure that you are appending data to the existing file, e.g. echo "Log entry" >> /var/log/nandan.log.

3

Thanks ,> Got your solution.
I was using vi editor fro inserting new data.
after that I tried to use echo method and it is working well.

I am just trying to find solution for real time log analysis.

Thanks

4

Logs written to by applications are typically appended to by default.

5

Yes correct.. So Filebeat will get data and new log data will go index automatically..
For this I checked as echo command..

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.