# 2 index same data **URL:** \ **Category:** Elasticsearch\ **Created:** [April 5, 2020, 6:56am UTC](https://discuss.elastic.co/t/2-index-same-data/226551 "2020-04-05T06:56:02Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![legolas\_bilbao](https://avatars.discourse-cdn.com/v4/letter/l/3da27b/32.png) [@legolas\_bilbao](https://discuss.elastic.co/u/legolas_bilbao)\ **Post date:** [April 5, 2020, 6:56am UTC](https://discuss.elastic.co/t/2-index-same-data/226551/1 "2020-04-05T06:56:02Z")
Hi everybody , I've two index defined in the same logstash and the origin of the data are two different tables of a database. When i read the data with logstash and paint them into kibana, the data is mixed , why does it happen? first index ``` input { jdbc { jdbc_connection_string => "jdbc:mysql://localhost:3306/eventos?useSSL=false" jdbc_user => "labo" jdbc_password => "arcsight_L4B0" jdbc_driver_library => "/mysql-connector-java-5.1.47.jar" jdbc_driver_class => "com.mysql.jdbc.Driver" schedule => "*/10 * * * *" statement => "SELECT * FROM eventos where ideventos>:sql_last_value" use_column_value => true clean_run => false tracking_column => ideventos jdbc_paging_enabled => true jdbc_page_size => 25000 } } output { elasticsearch { "hosts" => "localhost:9200" "index" => "arcsight" } stdout { codec => json_lines } } ``` second index > input { > jdbc { > jdbc\_connection\_string =\> "jdbc:mysql://localhost:3306/KPIS?useSSL=false" > jdbc\_user =\> "labo" > jdbc\_password =\> "arcsight\_L4B0" > jdbc\_driver\_library =\> "mysql-connector-java-5.1.47.jar" > jdbc\_driver\_class =\> "com.mysql.jdbc.Driver" > schedule =\> "46 8 \* \* 1,2,3,4,5,6,7" > statement =\> "Select \* FROM KPIS.reglasArcsight where idreglasArcsight\>:sql\_last\_value" > use\_column\_value =\> true > clean\_run =\> false > tracking\_column =\> "idreglasarcsight" > jdbc\_paging\_enabled =\> true > jdbc\_page\_size =\> 250000 > last\_run\_metadata\_path =\> "/usr/share/logstash/.logstash\_reglas\_jdbc\_last\_run" > } > } > output { > elasticsearch { > "hosts" =\> "localhost:9200" > "index" =\> "reglas" > } > stdout { codec =\> json\_lines } > } ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 5, 2020, 7:16am UTC](https://discuss.elastic.co/t/2-index-same-data/226551/2 "2020-04-05T07:16:29Z")
May be the index pattern in Kibana ? ---
**Author:** ![legolas\_bilbao](https://avatars.discourse-cdn.com/v4/letter/l/3da27b/32.png) [@legolas\_bilbao](https://discuss.elastic.co/u/legolas_bilbao)\ **Post date:** [April 5, 2020, 7:43am UTC](https://discuss.elastic.co/t/2-index-same-data/226551/3 "2020-04-05T07:43:16Z")
Arcsight pattern have the same fields as reglas pattern but the structure of the database is different. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 5, 2020, 7:57am UTC](https://discuss.elastic.co/t/2-index-same-data/226551/4 "2020-04-05T07:57:22Z")
If you put multiple files in the Logstash config directory these are concatenated into a single pipeline where all input data is processed by all filters and go to all outputs unless you control this through conditionals or use the multi pipeline feature. That is why both indices hold exactly the same data. This is a common misunderstanding and you should be able to find many examples in this forum under the Logstash category. ---
**Author:** ![legolas\_bilbao](https://avatars.discourse-cdn.com/v4/letter/l/3da27b/32.png) [@legolas\_bilbao](https://discuss.elastic.co/u/legolas_bilbao)\ **Post date:** [April 5, 2020, 8:37am UTC](https://discuss.elastic.co/t/2-index-same-data/226551/5 "2020-04-05T08:37:15Z")
Hi christian, I'll search for multipipeline many thanks ---
**Author:** ![legolas\_bilbao](https://avatars.discourse-cdn.com/v4/letter/l/3da27b/32.png) [@legolas\_bilbao](https://discuss.elastic.co/u/legolas_bilbao)\ **Post date:** [April 5, 2020, 8:57am UTC](https://discuss.elastic.co/t/2-index-same-data/226551/6 "2020-04-05T08:57:14Z")
I've use multipipile path.config: "/etc/logstash/conf.d/arcsight.conf" - pipeline.id: reglas path.config: "/etc/logstash/conf.d/reglas.conf" [2020-04-05T10:45:03,537][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"arcsight", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50} [2020-04-05T10:45:03,916][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"reglas", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50} [2020-04-05T10:45:04,511][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://localhost:9200/](http://localhost:9200/)]}} [2020-04-05T10:45:04,513][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://localhost:9200/](http://localhost:9200/)]}} [2020-04-05T10:45:04,898][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"} [2020-04-05T10:45:04,905][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"} [2020-04-05T10:45:05,019][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es\_version=\>6} [2020-04-05T10:45:05,021][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es\_version=\>6} [2020-04-05T10:45:05,022][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6} [2020-04-05T10:45:05,034][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6} [2020-04-05T10:45:05,096][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//localhost:9200](https://localhost:9200)"]} [2020-04-05T10:45:05,097][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//localhost:9200](https://localhost:9200)"]} [2020-04-05T10:45:05,128][INFO][logstash.outputs.elasticsearch] Using default mapping template [2020-04-05T10:45:05,143][INFO][logstash.outputs.elasticsearch] Using default mapping template [2020-04-05T10:45:05,182][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}} [2020-04-05T10:45:05,184][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}} [2020-04-05T10:45:05,519][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"arcsight", :thread=\>"#\"} [2020-04-05T10:45:05,521][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"reglas", :thread=\>"#\"} [2020-04-05T10:45:05,725][INFO][logstash.agent] Pipelines running {:count=\>2, :running\_pipelines=\>[:arcsight, :reglas], :non\_running\_pipelines=\>} But the index related to reglas doesnt appear. do i miss sth? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 5, 2020, 9:33am UTC](https://discuss.elastic.co/t/2-index-same-data/226551/7 "2020-04-05T09:33:48Z")
You need to specify both pipelines in pipeline.yml. To be sure these are being used I would recommend storing these config files outside the `conf.d` directory. Please share the full `pipelines.yml` file and make sure you format it properly. ---
**Author:** ![legolas\_bilbao](https://avatars.discourse-cdn.com/v4/letter/l/3da27b/32.png) [@legolas\_bilbao](https://discuss.elastic.co/u/legolas_bilbao)\ **Post date:** [April 5, 2020, 10:51am UTC](https://discuss.elastic.co/t/2-index-same-data/226551/8 "2020-04-05T10:51:07Z")
Fixed, many thanks!! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 3, 2020, 10:51am UTC](https://discuss.elastic.co/t/2-index-same-data/226551/9 "2020-05-03T10:51:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.