Duplicate logs In Kibana

[Shlok_Srivastava1 (Shlok Srivastava)]

I am getting Double logs for every api hits in Kibana.

Following is my logstash configuration assuming the issue is related to logstash

input {
 beats {
   port => 5044
  ssl => false
  }
}
           
output {
elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "%{[@metadata][type]}-%{+YYYY.MM.dd}"
  	document_type => "log"
 }
}

I fixed the above issue using fingerprints and now onwards I am not getting any duplicates. However old logs are the same i.e, they are having duplicates. Is there any way to reset them?

[Rob_wylde]

Can we also see the filebeat.yml file?

Also try removing "document_type" from your output as that option is deprecated. Is there a specific reason you've set manage_template to false?

Additionally why not just point filebeat directly at elastic if you're not doing any filtering in logstash?

This article may aid in the remove of duplicates. https://www.elastic.co/blog/how-to-find-and-remove-duplicate-documents-in-elasticsearch

[Shlok_Srivastava1 (Shlok Srivastava)]

###################### Filebeat Configuration Example 
#########################

filebeat.inputs:

- type: log

  enabled: true

  paths:
#- /var/log/*.log
#- /var/log/apache2/*.log
- /home/ubuntu/ProjectName/logs/*
#- c:\programdata\elasticsearch\logs\*


filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml


setup.template.settings:
  index.number_of_shards: 3


output.logstash:
  hosts: ["localhost:5044"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

Inside the location, I have multiple files resembling logs of each past day (eg log-31-12-2019) and one log of the current day. I have checked manually the duplication is not present here.

There isnt any specific reason for putting manage_template as false. I kinda copy pasted it.