# Problem with output elasticsearch data duplicate on index

**URL:** <https://discuss.elastic.co/t/problem-with-output-elasticsearch-data-duplicate-on-index/101981>\
**Category:** Logstash\
**Created:** [September 27, 2017, 1:02pm UTC](https://discuss.elastic.co/t/problem-with-output-elasticsearch-data-duplicate-on-index/101981 "2017-09-27T13:02:59Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![eldiablo62](https://avatars.discourse-cdn.com/v4/letter/e/c67d28/32.png) [@eldiablo62](https://discuss.elastic.co/u/eldiablo62)\
**Post date:** [September 27, 2017, 1:02pm UTC](https://discuss.elastic.co/t/problem-with-output-elasticsearch-data-duplicate-on-index/101981/1 "2017-09-27T13:02:59Z")

</div>

Hi,  
I have a problem, i have configured 2 datasources and i want to index the data different index but my data are not on the good index.

my /etc/logstash/conf.d/config\_mysql\_glpi\_tickets.conf

```
input {
jdbc {
    jdbc_driver_library => "/usr/share/logstash/vendor/jdbc-mysql/mysql-connector-java-5.1.44-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://ip:3306/glpi"
    jdbc_user => "user"
    jdbc_password => "pass"
    schedule => "*/5 * * * *"
    jdbc_page_size => 50000
    jdbc_paging_enabled => true
    statement => "SELECT * FROM view_tickets"
    use_column_value => true
    tracking_column => "id"
    tracking_column_type => "numeric"
}

```

}  
output {  
elasticsearch {  
hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
index =\> "glpi\_tickets"  
document\_type =\> "tickets"  
document\_id =\> "%{id}"  
}  
}

And my /etc/logstash/conf.d/config\_mysql\_glpi\_tasks.conf

```
input {
jdbc {
    jdbc_driver_library => "/usr/share/logstash/vendor/jdbc-mysql/mysql-connector-java-5.1.44-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://ip:3306/glpi"
    jdbc_user => "user"
    jdbc_password => "pass"
    schedule => "*/5 * * * *"
    jdbc_page_size => 50000
    jdbc_paging_enabled => true
    statement => "SELECT * FROM view_tickets"
    use_column_value => true
    tracking_column => "id"
    tracking_column_type => "numeric"
}

```

}  
output {  
elasticsearch {  
hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
index =\> "glpi\_tickets"  
document\_type =\> "tickets"  
document\_id =\> "%{id}"  
}  
}

On my glpi\_tickets index i have tasks document. I don't understand my mistake

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 27, 2017, 1:20pm UTC](https://discuss.elastic.co/t/problem-with-output-elasticsearch-data-duplicate-on-index/101981/2 "2017-09-27T13:20:03Z")

</div>

When you put multiple config files in a directory for Logstash to use, they will all be concatenated. This means that if you do not add a tag/field with each input and use [conditionals](https://www.elastic.co/guide/en/logstash/5.6/event-dependent-configuration.html#event-dependent-configuration) abided on this tag/field, all data will go to all output plugins.

---

<div class="post-metadata">

**Author:** ![eldiablo62](https://avatars.discourse-cdn.com/v4/letter/e/c67d28/32.png) [@eldiablo62](https://discuss.elastic.co/u/eldiablo62)\
**Post date:** [September 27, 2017, 1:32pm UTC](https://discuss.elastic.co/t/problem-with-output-elasticsearch-data-duplicate-on-index/101981/3 "2017-09-27T13:32:24Z")

</div>

hum, now i understand why 😉 But for solved can i put a value like this data\_type =\> tickets in my input block and do on if( data\_type == 'tickets') {} on my output block ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 27, 2017, 1:49pm UTC](https://discuss.elastic.co/t/problem-with-output-elasticsearch-data-duplicate-on-index/101981/4 "2017-09-27T13:49:54Z")

</div>

You can use the `add_field` parameter in the input to create a `data_type` field and then your conditional in the output section.

---

<div class="post-metadata">

**Author:** ![eldiablo62](https://avatars.discourse-cdn.com/v4/letter/e/c67d28/32.png) [@eldiablo62](https://discuss.elastic.co/u/eldiablo62)\
**Post date:** [September 27, 2017, 2:16pm UTC](https://discuss.elastic.co/t/problem-with-output-elasticsearch-data-duplicate-on-index/101981/5 "2017-09-27T14:16:04Z")

</div>

Thank for your help  
I share my config files for other people in same case

```
input {
jdbc {
jdbc_driver_library => "/usr/share/logstash/vendor/jdbc-mysql/mysql-connector-java-5.1.44-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://ip:3306/glpi"
jdbc_user => "user"
jdbc_password => "password"
schedule => "* * * * *"
jdbc_page_size => 50000
jdbc_paging_enabled => true
statement => "SELECT * FROM view_tickets"
use_column_value => true
tracking_column => "id"
tracking_column_type => "numeric"
add_field => { "data_source" => "glpi_tickets" }

```

}  
}  
output {  
if [data\_source] == "glpi\_tickets" {  
elasticsearch {  
hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
index =\> "glpi\_tickets"  
document\_type =\> "tickets"  
user =\> "elastic"  
password =\> "changeme"  
document\_id =\> "%{id}"  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 25, 2017, 2:16pm UTC](https://discuss.elastic.co/t/problem-with-output-elasticsearch-data-duplicate-on-index/101981/6 "2017-10-25T14:16:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.