# Log duplicated in multiple indexes

**URL:** <https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341>\
**Category:** Logstash\
**Created:** [August 18, 2016, 11:53am UTC](https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341 "2016-08-18T11:53:37Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\
**Post date:** [August 18, 2016, 11:53am UTC](https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341/1 "2016-08-18T11:53:37Z")

</div>

Hi,

In my output section, i have multiple conditions. Each conditions alows me to route one type of log in the correct Elasticsearch indexe.

This is my logstash output :

```
> output {
> if "nginx" in [tags] {
> if "_grokparsefailure" not in [tags] {
> elasticsearch {
> hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"]
> index => "logstash-isg-%{+YYYY.MM.dd}"
> }
> }
> }
> else if "scarlette" in [tags] {
> if "_grokparsefailure" not in [tags] {
> elasticsearch {
> hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"]
> index => "logstash-scarlette-%{+YYYY.MM.dd}"
> }
> }
> }
> else if "serveur_owncloud" in [tags] {
> if "_grokparsefailure" not in [tags] {
> elasticsearch {
> hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"]
> index => "logstash-owncloud-%{+YYYY.MM.dd}"
> }
> }
> }
> else if "brouette" in [tags] or "poussette" in [tags] {
> if "_grokparsefailure" not in [tags] {
> elasticsearch {
> hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"]
> index => "logstash-mta-%{+YYYY.MM.dd}"
> }
> }
> }
> else if "serveur_proxy" in [tags] or "serveur_dns" in [tags] {
> if "_grokparsefailure" not in [tags] {
> elasticsearch {
> hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"]
> index => "logstash-proxydns-%{+YYYY.MM.dd}"
> }

> # file {
> # path => "/var/log/LS-redis-flux.log"
> # }
> }
> }
> }

```

The spécified indexes are created, and they do contain the log i want.

But Logstash continue to create it's default indexe "logstash-_", and this indexe contain a copy of each log. One log is in "logstash-_" and the other is in the correct indexe.

This seems strange to me, because i always specifies the name of the indexe. in my logstash output conditions. I don't know why this indexe is created and why logs are copied into it.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [August 18, 2016, 2:31pm UTC](https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341/2 "2016-08-18T14:31:54Z")

</div>

Try without the else clause and swap the grokparsefailure logic.

e.g.

```auto
output {
    if "_grokparsefailure" not in [tags] {
        if "nginx" in [tags] {
                elasticsearch {
                        hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"]
                        index => "logstash-isg-%{+YYYY.MM.dd}"
                }
        }

        if "scarlette" in [tags] {
                elasticsearch {
                        hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"]
                        index => "logstash-scarlette-%{+YYYY.MM.dd}"
                }
        }

        if "serveur_owncloud" in [tags] {
                elasticsearch {
                        hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"]
                        index => "logstash-owncloud-%{+YYYY.MM.dd}"
                }
        }

        if "brouette" in [tags] or "poussette" in [tags] {
                elasticsearch {
                        hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"]
                        index => "logstash-mta-%{+YYYY.MM.dd}"
                }
        }

        if "serveur_proxy" in [tags] or "serveur_dns" in [tags] {
                elasticsearch {
                        hosts => ["10.1.101.1", "10.1.102.1", "10.1.103.1"]
                        index => "logstash-proxydns-%{+YYYY.MM.dd}"
                }

# file {
# path => "/var/log/LS-redis-flux.log"
# }
        }
    }
}

```

---

<div class="post-metadata">

**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\
**Post date:** [August 19, 2016, 7:21am UTC](https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341/3 "2016-08-19T07:21:29Z")

</div>

Thanks for the reply,

Unfortunatly, this changes in the configuration file of Logstash, did not change anything...

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [August 19, 2016, 7:29am UTC](https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341/4 "2016-08-19T07:29:34Z")

</div>

Please state what versions of logstash and the elasticsearch output you are using.

---

<div class="post-metadata">

**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\
**Post date:** [August 19, 2016, 7:54am UTC](https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341/5 "2016-08-19T07:54:06Z")

</div>

I use Logstash 2.2.0

I dont know how to see the version of the logstash-output-elasticsearch plugin. When i'm in the /opt/logstash directory i just can install, uninstall, update, pack, unpack or list plugins. But when i list all the installed plugins, it just give me the name of plugins not the version....

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [August 19, 2016, 12:33pm UTC](https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341/6 "2016-08-19T12:33:29Z")

</div>

use `bin/plugin list --verbose`

---

<div class="post-metadata">

**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\
**Post date:** [August 19, 2016, 12:47pm UTC](https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341/7 "2016-08-19T12:47:18Z")

</div>

Thanks,

So i use logstash 2.2.0 and logstas-output-elasticsearch 2.4.1

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [August 19, 2016, 1:03pm UTC](https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341/8 "2016-08-19T13:03:02Z")

</div>

When you run Logstash do you specify a file or a directory for the `-f` option?

```auto
    -f, --config CONFIG_PATH Load the logstash config from a specific file
                                  or directory. If a directory is given, all
                                  files in that directory will be concatenated
                                  in lexicographical order and then parsed as a
                                  single config file. You can also specify
                                  wildcards (globs) and any matched files will
                                  be loaded in the order described above.

```

---

<div class="post-metadata">

**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\
**Post date:** [August 19, 2016, 1:10pm UTC](https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341/9 "2016-08-19T13:10:28Z")

</div>

I run Logstash as a service and i don't specify a file or directory, so by default it use my config file "logstash.conf" in the conf.d directory.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [August 19, 2016, 2:08pm UTC](https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341/10 "2016-08-19T14:08:59Z")

</div>

I have looked at the code for elasticsearch-output 2.4.1.  
Hypothesis: it is retrying and the index get reset to the default on the second try

For this hypothesis to not be true, you should not see any log messages with this text in the log line `retrying failed action with response code`

Are you able to see the logstash logs? if so is there a warning level line with the above text in it?

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [August 19, 2016, 2:14pm UTC](https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341/11 "2016-08-19T14:14:46Z")

</div>

Are you able to see the Elasticsearch logs? What does it show about the http bulk index request?  
Can you see if the duplicates all come after all the originals were done or are they interleaved?

---

<div class="post-metadata">

**Author:** ![Clement\_Ros](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clement_ros/32/6198_2.png) [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\
**Post date:** [August 22, 2016, 7:41am UTC](https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341/12 "2016-08-22T07:41:31Z")

</div>

I'm not able to see the log line "retrying failed action with response code" , in the logstash log file. I haven't any warning log.

I whatched the Elasticsearch logs, and i didn't see any log about http bulk index request...

The duplicates logs come in the same time than the originals.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:42am UTC](https://discuss.elastic.co/t/log-duplicated-in-multiple-indexes/58341/13 "2017-07-06T04:42:23Z")

</div>