Duplicate entries for same data in multiple indices

Elastic Stack Logstash
1

Hello All,
I'm using logstash (5.3) to parse data and send it to elastic search.
I currently have 3 configuration files under config.d which parses 3 different types of data (all data is being collected from a database and sent to elastic search after getting parsed) and each conf file has its own index where the output is written to.

I'm trying to parse a new .csv file and send it to elastic search and have created a new index for it. I see that the data is being parsed and is being sent to the new index in kibana.
But I also see the data entries from other 3 indices in the new index as well. I see the data for other 3 index in both their original index and the new index created for the .csv files.

Below is the sample conf file I'm using to parse .csv file :

input {
file {
path => "/home/Amarender.KasiReddy/*.csv"
}
}
filter {
csv {
separator => ","
columns => ["ErrorThreshold","AppStatus","AgentStatus"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "servermonitor-%{+YYYY.MM.dd}"
}
}

Can someone please help me with this?

2

I currently have 3 configuration files under config.d which parses 3 different types of data (all data is being collected from a database and sent to Elasticsearch after getting parsed) and each conf file has its own index where the output is written to.

All configuration files are effectively merged together. Hence, all filters apply to all events from all inputs and will then be sent to all outputs. If this is undesirable you have three choices:

  • Use the multi-pipeline feature (introduced in Logstash 6).
  • Use conditionals to select which filters to apply to which events and which events to send to which outputs.
  • Run multiple instances of Logstash.
3

Thank you for the response @magnusbaeck.
I added the document_id to give a unique id for the data instead of having default id and I don't see the duplicate entries now. :slight_smile:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.