Duplicate Data in two indexes

Elastic Stack Logstash
1

Hi,

I am using logstash version 1.4.2

I configure logstash to store data in elasticsearch in a manual index name "xyz" and I am using following template for this index.

{
"template" : "xyz-",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true, "omit_norms" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true
}
}
}, {
"string_fields" : {
"match" : "",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "not_analyzed", "omit_norms" : true,
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
}
}
}
} ],
"properties" : {
"@version": { "type": "string", "index": "not_analyzed" },
"geoip" : {
"type" : "object",
"dynamic": true,
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
}
}

Issue is logstash storing data both in default index "logstash" and manual index "xyz".

I want to store data only in one index "xyz".

I am using "elasticsearch_http" output plugin to store data with following configuration.

elasticsearch_http {
host => "127.0.0.1"
index => "xyz-%{+YYYY.MM.dd}"
template => "/opt/logstash/lib/logstash/outputs/elasticsearch/elasticsearch-xyz.json"
template_overwrite => true
template_name => "xyz"
}

2

But you also have an elasticsearch_http output for storing events in the default "logstash" series of indexes, right?

3

In output plugin elasticsearch_http, if index is not defined then in that case plugin will store data in logstash index. But i configured my output plugin to store data in xyz index.

Still logstash storing data in both indexes xyz and logstash.

4

In that case I'd say you have another elasticsearch_http output, perhaps in another file, that doesn't override the index option. Look in /etc/logstash/conf.d and remember that Logstash reads all files there. You can also start Logstash with --debug to see exactly which configuraion is being loaded.

5

I didn't find exact root cause for this, however I find a workaround for this. I configured if condition in output plugin and it's working for me.

output {
    elasticsearch_http {
        if [type] == "xyz" { 
            host => "127.0.0.1"
            index => "xyz-%{+YYYY.MM.dd}"
            template => "/opt/logstash/lib/logstash/outputs/elasticsearch/elasticsearch-xyz.json"
            template_overwrite => true
            template_name => "xyz"
        }
    }
}