Kibana and index patterns performance

#1

Hi all,

I'm collecting information with filebeat, procesing it with logstash and storing in elasticsearch. I have different types and sub-types of information, and indexing the information with something similar to
%{[type]}-%{sub_type}-%{+YYYY.MM.dd}

For the same "type" i have several "sub_type".

Then we have one dashboard with several charts and tables getting data from those indexes.

The question is regarding performance, what is the best option:

  1. having one index pattern type* and then in the charts adding filters for sub_type.keyword:XX for the different sub_type

or

  1. having one index pattern for each sub_type: type-sub_type* and in this case not using the filter

thanks

#2

Hi

Assuming you have a large amount of data to query, the second pattern should be more performant, because there will be less indices queried.

pattern 1:
your-type1-.
pattern 2:
your-type1-subtype1.
your-type1-subtype2*
...
With pattern2 you will basically pre-filter you data. You won't notice much difference at a small amount of data, but at large scale this should have a better performance

1 Like
#3

Thanks Matthias, and in case of second patter with indexes like:

%{[type]}-%{sub_type}-%{+YYYY.MM.dd}

When you filter by dates in dashboards, does it go all over all indexes or kibana knows the index format and go over the related indexes?

thanks

#4

In this case there is no further optimization by the index name, like eg you define a range of a month, you want kibana just to use

your-type1-subtype2-2019-12-*

that's currently not possible

1 Like