# Duplicate Data in two indexes

**URL:** <https://discuss.elastic.co/t/duplicate-data-in-two-indexes/36869>\
**Category:** Logstash\
**Created:** [December 10, 2015, 11:41am UTC](https://discuss.elastic.co/t/duplicate-data-in-two-indexes/36869 "2015-12-10T11:41:35Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gurveer\_Singh](https://avatars.discourse-cdn.com/v4/letter/g/f6c823/32.png) [@Gurveer\_Singh](https://discuss.elastic.co/u/Gurveer_Singh)\
**Post date:** [December 10, 2015, 11:41am UTC](https://discuss.elastic.co/t/duplicate-data-in-two-indexes/36869/1 "2015-12-10T11:41:35Z")

</div>

Hi,

I am using logstash version 1.4.2

I configure logstash to store data in elasticsearch in a manual index name **"xyz"** and I am using following template for this index.

> {  
> "template" : "xyz-_",  
> "settings" : {  
> "index.refresh\_interval" : "5s"  
> },  
> "mappings" : {  
> "default" : {  
> "\_all" : {"enabled" : true, "omit\_norms" : true},  
> "dynamic\_templates" : [ {  
> "message\_field" : {  
> "match" : "message",  
> "match\_mapping\_type" : "string",  
> "mapping" : {  
> "type" : "string", "index" : "analyzed", "omit\_norms" : true  
> }  
> }  
> }, {  
> "string\_fields" : {  
> "match" : "_",  
> "match\_mapping\_type" : "string",  
> "mapping" : {  
> "type" : "string", "index" : "not\_analyzed", "omit\_norms" : true,  
> "fields" : {  
> "raw" : {"type": "string", "index" : "not\_analyzed", "ignore\_above" : 256}  
> }  
> }  
> }  
> } ],  
> "properties" : {  
> "@version": { "type": "string", "index": "not\_analyzed" },  
> "geoip" : {  
> "type" : "object",  
> "dynamic": true,  
> "properties" : {  
> "location" : { "type" : "geo\_point" }  
> }  
> }  
> }  
> }  
> }  
> }

Issue is logstash storing data both in default index **"logstash"** and manual index **"xyz"**.

I want to store data only in one index **"xyz"**.

I am using **"elasticsearch\_http"** output plugin to store data with following configuration.

> elasticsearch\_http {  
> host =\> "127.0.0.1"  
> index =\> "xyz-%{+YYYY.MM.dd}"  
> template =\> "/opt/logstash/lib/logstash/outputs/elasticsearch/elasticsearch-xyz.json"  
> template\_overwrite =\> true  
> template\_name =\> "xyz"  
> }

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 10, 2015, 12:37pm UTC](https://discuss.elastic.co/t/duplicate-data-in-two-indexes/36869/2 "2015-12-10T12:37:14Z")

</div>

But you also have an elasticsearch\_http output for storing events in the default "logstash" series of indexes, right?

---

<div class="post-metadata">

**Author:** ![Gurveer\_Singh](https://avatars.discourse-cdn.com/v4/letter/g/f6c823/32.png) [@Gurveer\_Singh](https://discuss.elastic.co/u/Gurveer_Singh)\
**Post date:** [December 10, 2015, 1:04pm UTC](https://discuss.elastic.co/t/duplicate-data-in-two-indexes/36869/3 "2015-12-10T13:04:45Z")

</div>

In output plugin elasticsearch\_http, if index is not defined then in that case plugin will store data in **logstash** index. But i configured my output plugin to store data in **xyz** index.

Still logstash storing data in both indexes **xyz** and **logstash**.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 10, 2015, 1:21pm UTC](https://discuss.elastic.co/t/duplicate-data-in-two-indexes/36869/4 "2015-12-10T13:21:49Z")

</div>

In that case I'd say you have another elasticsearch\_http output, perhaps in another file, that doesn't override the `index` option. Look in /etc/logstash/conf.d and remember that Logstash reads _all_ files there. You can also start Logstash with `--debug` to see exactly which configuraion is being loaded.

---

<div class="post-metadata">

**Author:** ![Gurveer\_Singh](https://avatars.discourse-cdn.com/v4/letter/g/f6c823/32.png) [@Gurveer\_Singh](https://discuss.elastic.co/u/Gurveer_Singh)\
**Post date:** [December 31, 2015, 12:37pm UTC](https://discuss.elastic.co/t/duplicate-data-in-two-indexes/36869/5 "2015-12-31T12:37:36Z")

</div>

I didn't find exact root cause for this, however I find a workaround for this. I configured **if** condition in output plugin and it's working for me.

```
output {
    elasticsearch_http {
        if [type] == "xyz" { 
            host => "127.0.0.1"
            index => "xyz-%{+YYYY.MM.dd}"
            template => "/opt/logstash/lib/logstash/outputs/elasticsearch/elasticsearch-xyz.json"
            template_overwrite => true
            template_name => "xyz"
        }
    }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:17am UTC](https://discuss.elastic.co/t/duplicate-data-in-two-indexes/36869/6 "2017-07-06T05:17:07Z")

</div>