Kibana4 slow response using weekly indexes

Elastic Stack Kibana
1

Hello,

We are using Kibana 4.3.1 with Elasticsearch 2.1.0 and considering moving from a single index (5 shards) of 40gb to partition the index on per week basis i.e. 52 weekly indices (1 shard) holding varing size 0.2 - 1gb of data. The idea behind doing this was we could easily zap out the weekly indices which are not needed depending on for how long we want to keep the data.

The problem though is Kibana is when you create a search on the dashboard spanning 1month - 6 months (4-26 weeks) on the discover tab the data seems to be getting slowly and it appears like Kibana is firing multiple queries to retrieve results from all indices even though we are using an index pattern like "myindex-*". This is far slow compared to a single index which just gets data quickly.

However there are certain visualizations like we have one for Metrics which even though spanning across 26 weeks of data, get results faster compared to our single index approach.

It appears like Kibana 4 with wildcard index pattern on weekly partitioned index responds slower if it has to get more data (i.e. discover tab) but responds quickly with less data being returned from query (i.e. aggregations for e.g. in metrics).

Overall though we see the queries on ES with wildcard index respond quickly in partitioned approach but Kibana 4 discover tab is may be making multiple queries to get data.

Any ideas why this is the case and any possible resolutions? Should we try something differently with the index pattern in Kibana ?

N.B. We chose "Index contains time-based events" whilst creating the index pattern myindex-* and did not choose "Use event times to create index names" since that seems to be marked as Deprecated in Kibana 4.3.1

2

I think found a solution (or workaround may be) to this which is to create an index alias and point it to look at the index partitions i.e. an alias like myindexalias on myindex-*.

And then in Kibana instead of using the index pattern as myindex-* simply use the alias i.e. myindexalias as the index. That makes Kibana to fire single search query to the alias in the Discover tab (or for searches) and the results seem to come back faster since now it is not making multiple queries and updating the results on the view.

Only thing to take care of is that when creating alias to myindex* it includes the myindex partitions at that point in time which means the alias needs to be refreshed when there are partition additions or deletions.

That works! But let me know if somebody has other ideas around this pls. ?