# To multiple index or not to multiple index...that is the question **URL:** \ **Category:** Elasticsearch\ **Created:** [January 26, 2017, 3:01pm UTC](https://discuss.elastic.co/t/to-multiple-index-or-not-to-multiple-index-that-is-the-question/72912 "2017-01-26T15:01:36Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Post date:** [January 26, 2017, 3:01pm UTC](https://discuss.elastic.co/t/to-multiple-index-or-not-to-multiple-index-that-is-the-question/72912/1 "2017-01-26T15:01:36Z")
I currently have the following: - syslog data coming in from network devices. - syslog data coming from applications - syslog data coming in from unix servers - syslog data coming in from ESX hosts - winlogbeat data from windows event logs Currently all these sources get collated into ONE index. I am looking into running Packetbeat on domain controllers for DNS logging as well as a couple other beats for metrics, etc. My question is, in order to view them easily in Kibana, I had put them all in the same index. Is there any value/performance increase in splitting these out into their own indexes? ---
**Author:** ![JKhondhu](https://avatars.discourse-cdn.com/v4/letter/j/ed655f/32.png) [@JKhondhu](https://discuss.elastic.co/u/JKhondhu)\ **Post date:** [January 26, 2017, 5:00pm UTC](https://discuss.elastic.co/t/to-multiple-index-or-not-to-multiple-index-that-is-the-question/72912/2 "2017-01-26T17:00:55Z")
_It depends._ How often does this one index rotate? What length of queries run upon these indices - Do they span for one week, one month? ---
**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Post date:** [January 26, 2017, 5:33pm UTC](https://discuss.elastic.co/t/to-multiple-index-or-not-to-multiple-index-that-is-the-question/72912/3 "2017-01-26T17:33:00Z")
Currently, they rotate daily. Here are some stats that may answer your questions in order to guide me better. Also, keep in mind, I just rebuilt this cluster from scratch. So it only has a few days worth of data, but the retention period is about 20 days. Eventually I will get more space to increase the retention, but this all I have at the moment. ``` { "cluster_name" : "########", "status" : "green", "timed_out" : false, "number_of_nodes" : 8, "number_of_data_nodes" : 4, "active_primary_shards" : 34, "active_shards" : 41, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } master name version role disk.avail heap.max ram.max ram.current cpu uptime jdk - DATANODE-01 5.1.2 di 1.6tb 11.9gb 15.6gb 15.5gb 31 1.7d 1.8.0_121 - DATANODE-02 5.1.2 di 1.6tb 11.9gb 15.6gb 15.5gb 46 1.7d 1.8.0_121 - DATANODE-03 5.1.2 di 1.6tb 11.9gb 15.6gb 15.2gb 51 1.7d 1.8.0_121 - DATANODE-04 5.1.2 di 1.6tb 11.9gb 15.6gb 15.4gb 67 1.7d 1.8.0_121 - KBNANODE-01 5.1.2 - 38.9gb 11.9gb 15.6gb 14.3gb 1 1.7d 1.8.0_121 - MSTRNODE-01 5.1.2 mi 39.6gb 11.9gb 15.6gb 13.5gb 2 1.7d 1.8.0_121 * MSTRNODE-02 5.1.2 mi 39.6gb 11.9gb 15.6gb 13.5gb 0 1.7d 1.8.0_121 - MSTRNODE-03 5.1.2 mi 39.6gb 11.9gb 15.6gb 13.5gb 0 1.7d 1.8.0_121 health status index green open logstash-2017.01.23 green open logstash-2017.01.24 green open logstash-2017.01.25 green open logstash-2017.01.26 index shard prirep state ip node logstash-2017.01.23 2 p STARTED x.x.x.x DATANODE-01 logstash-2017.01.23 1 p STARTED x.x.x.x DATANODE-02 logstash-2017.01.23 3 p STARTED x.x.x.x DATANODE-03 logstash-2017.01.23 0 p STARTED x.x.x.x DATANODE-04 logstash-2017.01.24 2 p STARTED x.x.x.x DATANODE-01 logstash-2017.01.24 1 p STARTED x.x.x.x DATANODE-02 logstash-2017.01.24 3 p STARTED x.x.x.x DATANODE-03 logstash-2017.01.24 0 p STARTED x.x.x.x DATANODE-04 logstash-2017.01.25 2 p STARTED x.x.x.x DATANODE-04 logstash-2017.01.25 1 p STARTED x.x.x.x DATANODE-03 logstash-2017.01.25 3 p STARTED x.x.x.x DATANODE-02 logstash-2017.01.25 0 p STARTED x.x.x.x DATANODE-01 logstash-2017.01.26 2 p STARTED x.x.x.x DATANODE-01 logstash-2017.01.26 1 p STARTED x.x.x.x DATANODE-02 logstash-2017.01.26 3 p STARTED x.x.x.x DATANODE-03 logstash-2017.01.26 0 p STARTED x.x.x.x DATANODE-04 shards disk.indices disk.used disk.avail disk.total disk.percent host ip node 10 225.7gb 328.9gb 1.6tb 1.9tb 16 x.x.x.x x.x.x.x DATANODE-01 10 225.6gb 328.4gb 1.6tb 1.9tb 16 x.x.x.x x.x.x.x DATANODE-02 11 227.1gb 330.3gb 1.6tb 1.9tb 16 x.x.x.x x.x.x.x DATANODE-03 10 227.7gb 330.5gb 1.6tb 1.9tb 16 x.x.x.x x.x.x.x DATANODE-04 master name indexing.index_total indexing.index_current indexing.index_failed indexing.delete_total - DATANODE-01 58864840 0 0 0 - DATANODE-02 58791438 0 0 0 - DATANODE-03 58715893 0 38 0 - DATANODE-04 58784564 0 0 0 - KBNANODE-01 0 0 0 0 - MSTRNODE-01 0 0 0 0 * MSTRNODE-02 0 0 0 0 - MSTRNODE-03 0 0 0 0 ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 26, 2017, 7:42pm UTC](https://discuss.elastic.co/t/to-multiple-index-or-not-to-multiple-index-that-is-the-question/72912/4 "2017-01-26T19:42:03Z")
You wouldn't put all that into a single DB table would you? So with that logic it makes sense to split stuff out, keep it hygienic, prevents mapping explosions, allows custom retention per source. ---
**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Post date:** [January 26, 2017, 9:11pm UTC](https://discuss.elastic.co/t/to-multiple-index-or-not-to-multiple-index-that-is-the-question/72912/5 "2017-01-26T21:11:31Z")
Hahah, well, you are talking to someone who knows just above 0.00001% of what he is doing or databases in general. But I will split them out. In terms of Kibana, is there an easy way to search multiple indexes in the Discover tab? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 26, 2017, 9:26pm UTC](https://discuss.elastic.co/t/to-multiple-index-or-not-to-multiple-index-that-is-the-question/72912/6 "2017-01-26T21:26:56Z")
Only what is in index patterns, so having a similar naming structure, eg `logstash-$type-$date`, will help. ---
**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Post date:** [January 26, 2017, 10:17pm UTC](https://discuss.elastic.co/t/to-multiple-index-or-not-to-multiple-index-that-is-the-question/72912/7 "2017-01-26T22:17:37Z")
So like: syslog data coming in from network devices - logstash-network-$date syslog data coming from applications - logstash-application-$date syslog data coming in from unix servers - logstash-unix-$date syslog data coming in from ESX hosts - logstash-esx-$date winlogbeat data from windows event logs - logstash-winlogbeat-$date ? For the most part, 99% of all the stored data is system logs, connection logs, event logs, etc. The reason I have them all in the same index was so I could punch in an IP address and see all the sources that referenced it. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 26, 2017, 10:35pm UTC](https://discuss.elastic.co/t/to-multiple-index-or-not-to-multiple-index-that-is-the-question/72912/8 "2017-01-26T22:35:33Z")
Yep like that. Then if you use a `logstash-*` index pattern in KB, you can still see all the data, but split things out. ---
**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Post date:** [January 26, 2017, 10:39pm UTC](https://discuss.elastic.co/t/to-multiple-index-or-not-to-multiple-index-that-is-the-question/72912/9 "2017-01-26T22:39:38Z")
Awesome sauce!!!!! I can't wait to give this a try. In terms of performance, do you think it will be a boost or a negative? ---
**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\ **Post date:** [January 27, 2017, 10:08pm UTC](https://discuss.elastic.co/t/to-multiple-index-or-not-to-multiple-index-that-is-the-question/72912/10 "2017-01-27T22:08:32Z")
> [@warkolm](#): > > ep like that.Then if you use a logstash-\* index pattern in KB, you can still see all the data, but split things out. With this pattern that includes all indexes of different types, when we filter a type in Kibana, will only the indexes that has this type be queried or will all indexes be queried? Also, will Kibana complain about data type conflict if the same field of different types in different indexes have different data type? e.g, ```auto logstash-index1 > type1 > my_field (string) logstash-index2 > type2 > my_field (long) ``` ---
**Author:** ![kopacko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kopacko/32/10733_2.png) [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Post date:** [January 31, 2017, 4:31pm UTC](https://discuss.elastic.co/t/to-multiple-index-or-not-to-multiple-index-that-is-the-question/72912/11 "2017-01-31T16:31:29Z")
So, I have one data center cluster using multiple indexes and one cluster using a single index. From what I can tell, the multi-index cluster seems to be missing some logs (not many) and is definitely SLOWER compared to the single index cluster. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 28, 2017, 4:31pm UTC](https://discuss.elastic.co/t/to-multiple-index-or-not-to-multiple-index-that-is-the-question/72912/12 "2017-02-28T16:31:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.