Will many narrow index patterns improve search performance?

Elastic Stack Elasticsearch
1

We're planning to migrate from single daily index to many indices in order to implement granular access policies.
I.e. instead of logstash-DD.MM.YYYY we're going to have about 40 daily indices like logstash-applicaitonXX-DD.MM.YYYY.
Should we add new index templates per application in addition to the default "logstash-*" in order to avoid scanning unnecessary indices?

TIA,
Vitaly

2

How much data are you ingesting per day? What is your retention period?

3

We're running Elastic on a standalone node with 16GB RAM, with 30 days retention, we have about 1300 indices, about 800GB data.

4

That sounds like a very large number of indices and shards given the cluster size and data volume. The default Logstash index template applies to indices starting with logstash-, so would apply to all indices following the naming convention you outlined.

5

Thank you!
So if we'll create new index templates ("logstash-nginx-*" and so on) it should decrease number of indices involved in search?
Or there is no choice and we should decrease number of indices?

6

The index template does not affect which indices that are included in a search, only how the data is mapped. If the current Logstash mappings are fine and you do not need to optimise mappings per application, you can continue using the default one.

In order to limit the indices being queried, you need to create more specific index patterns in Kibana and use these when you build dashboards. This also applies if you are querying in other ways. Having that many indices and shards is going to result in overhead, as each shard requires memory and file handles, so it is not a given that your query performance will improve, especially if you need to hit a lot of shards.

7

Of course, I meant Kibana index patterns.
Thank you again, we'll try to play with our configuration.
Vitaly

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.