# Why is it merging my indexes? **URL:** \ **Category:** Logstash\ **Created:** [March 21, 2018, 5:18pm UTC](https://discuss.elastic.co/t/why-is-it-merging-my-indexes/125048 "2018-03-21T17:18:20Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![hgpit](https://avatars.discourse-cdn.com/v4/letter/h/f1d935/32.png) [@hgpit](https://discuss.elastic.co/u/hgpit)\ **Post date:** [March 21, 2018, 5:18pm UTC](https://discuss.elastic.co/t/why-is-it-merging-my-indexes/125048/1 "2018-03-21T17:18:20Z")
Hello, I am experimenting with Logstash 6. I have two \*.conf files. One is sending results to index " **sshd\_fail-%{+YYYY.MM}**", the second is sending results to index " **idx\_md-descriptions**". I have installed Kibana, and I attempt to create an index pattern. I choose my index as 'sshd\_fail-\*', and it shows all the available fields. Problem is that in the list of fields is also all the fields from my other index. Elasticsearch shows my sshd\_fail index to be huge: [root@svr-h000386 incomingdata]# curl 10.11.2.11:9200/\_cat/indices?v health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open .kibana eBNdoaSzTXm2OqZJ8vDvgw 1 0 2 1 11kb 11kb yellow open sshd\_fail-2018.03 Lnwzz5BnTr2HjXokB6rCHA 5 1 2275996 0 721.9mb 721.9mb yellow open idx\_md-descriptions XNgMSo9gScewN7BCwkqslw 5 1 1321214 166304 554.5mb 554.5mb Why is Logstash apparently merging my two data sources into both indexes? **_FILE1.conf_** input { jdbc { jdbc\_connection\_string =\> "jdbc:mysql://svr-h003671.my-domain.local:3306/mdata" jdbc\_user =\> "elastic" jdbc\_password =\> "blah" jdbc\_driver\_library =\> "/usr/share/java/mysql-connector-java.jar" jdbc\_driver\_class =\> "com.mysql.jdbc.Driver" jdbc\_paging\_enabled =\> "true" jdbc\_page\_size =\> "50000" schedule =\> "5 \* \* \* \*" statement =\> "SELECT item\_code,item\_description,brand\_name FROM tbl\_products p LEFT JOIN tbl\_brands b ON b.brand\_id = p.brand\_id" } } output { elasticsearch { hosts =\> ["10.11.2.11:9200"] index =\> "idx\_md-descriptions" document\_id =\> "%{item\_code}" } } **FILE2.conf** input { file { type =\> "secure\_log" path =\> "/var/log/secure" } } filter { grok { add\_tag =\> ["sshd\_fail"] match =\> { "message" =\> "Failed %{WORD:sshd\_auth\_type} for %{USERNAME:sshd\_invalid\_user} from %{IP:sshd\_client\_ip} port %{NUMBER:sshd\_port} %{GREEDYDATA:sshd\_protocol}" } } } output { elasticsearch { hosts =\> ["10.11.2.11:9200"] index =\> "sshd\_fail-%{+YYYY.MM}" } } ---
**Author:** ![hgpit](https://avatars.discourse-cdn.com/v4/letter/h/f1d935/32.png) [@hgpit](https://discuss.elastic.co/u/hgpit)\ **Post date:** [March 21, 2018, 5:52pm UTC](https://discuss.elastic.co/t/why-is-it-merging-my-indexes/125048/2 "2018-03-21T17:52:46Z")
Hello, I have tried to split out my input and output, using conditionals, but now my database index isn't working at all: ``` input { jdbc { jdbc_connection_string => "jdbc:mysql://svr-h003671.hayley-group.local:3306/masdata" jdbc_user => "elastic" jdbc_password => "iGr0up!T" jdbc_driver_library => "/usr/share/java/mysql-connector-java.jar" jdbc_driver_class => "com.mysql.jdbc.Driver" jdbc_paging_enabled => "true" jdbc_page_size => "50000" schedule => "5 * * * *" statement => "SELECT item_code,item_description,brand_name FROM tbl_products p LEFT JOIN tbl_brands b ON b.brand_id = p.brand_id" tags => "idx-md_descriptions" } file { tags => "secure_log" path => "/var/log/secure" } } output { if "idx-md_descriptions" in [tags] { elasticsearch { hosts => ["10.11.2.11:9200"] index => "idx-md_descriptions" document_id => "%{item_code}" } } else if "secure_log" in [tags] { elasticsearch { hosts => ["10.11.2.11:9200"] index => "sshd_fail-%{+YYYY.MM}" } } } ``` Only one index being reported by ES: ``` [root@svr-h000386 conf.d]# curl 10.11.2.11:9200/_cat/indices?v health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open .kibana eBNdoaSzTXm2OqZJ8vDvgw 1 0 2 1 11kb 11kb yellow open sshd_fail-2018.03 RGIrgbvxTaSnCOYG8yXJhA 5 1 6 0 27.5kb 27.5kb [root@svr-h000386 conf.d]# ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 21, 2018, 6:27pm UTC](https://discuss.elastic.co/t/why-is-it-merging-my-indexes/125048/3 "2018-03-21T18:27:37Z")
That looks like the right approach. Do you get any indexing errors in the logstash logs? Can you add a stdout { codec =\> rubydebug } output and see what one of those idx-md\_descriptions events looks like? You did wait for the schedule of the jdbc input to trigger, right? ---
**Author:** ![hgpit](https://avatars.discourse-cdn.com/v4/letter/h/f1d935/32.png) [@hgpit](https://discuss.elastic.co/u/hgpit)\ **Post date:** [March 22, 2018, 9:05am UTC](https://discuss.elastic.co/t/why-is-it-merging-my-indexes/125048/4 "2018-03-22T09:05:03Z")
Hi Badger, Thanks for your reply. I was working on this for hours yesterday, more than enough time for the 5 minute window... but nothing... until this morning! I have got in the office today to find the DB index has now built. Weird. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 22, 2018, 12:31pm UTC](https://discuss.elastic.co/t/why-is-it-merging-my-indexes/125048/5 "2018-03-22T12:31:29Z")
> [@hgpit](#): > > more than enough time for the 5 minute window It's not a [5 minute window](https://discuss.elastic.co/t/how-to-run-the-schedule-every-five-minutes-in-logstash-5-0/66222/2), that cron schedule runs at 5 minutes past each hour. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 19, 2018, 12:31pm UTC](https://discuss.elastic.co/t/why-is-it-merging-my-indexes/125048/6 "2018-04-19T12:31:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.