Kibana and index patterns performance

Hi all,

I'm collecting information with filebeat, procesing it with logstash and storing in elasticsearch. I have different types and sub-types of information, and indexing the information with something similar to
%{[type]}-%{sub_type}-%{+YYYY.MM.dd}

For the same "type" i have several "sub_type".

Then we have one dashboard with several charts and tables getting data from those indexes.

The question is regarding performance, what is the best option:

  1. having one index pattern type* and then in the charts adding filters for sub_type.keyword:XX for the different sub_type

or

  1. having one index pattern for each sub_type: type-sub_type* and in this case not using the filter

thanks

Hi

Assuming you have a large amount of data to query, the second pattern should be more performant, because there will be less indices queried.

pattern 1:
your-type1-.
pattern 2:
your-type1-subtype1
.
your-type1-subtype2*
...
With pattern2 you will basically pre-filter you data. You won't notice much difference at a small amount of data, but at large scale this should have a better performance

1 Like

Thanks Matthias, and in case of second patter with indexes like:

%{[type]}-%{sub_type}-%{+YYYY.MM.dd}

When you filter by dates in dashboards, does it go all over all indexes or kibana knows the index format and go over the related indexes?

thanks

In this case there is no further optimization by the index name, like eg you define a range of a month, you want kibana just to use

your-type1-subtype2-2019-12-*

that's currently not possible

1 Like