Performance of Multiple index patterns vs. a single index pattern

Elastic Stack Elasticsearch
1

Hi all,

I have 28 types of logs which total up to hundreds of millions event per hour. The indexes are named as following:

  • log_type_1-
  • log_type_2-
  • log_type_3-
  • log_type_n-

For search and aggregation, is there performance difference between using an single log_* index pattern vs. creating an index pattern for each log type (e.g., log_type_1-*)?

If I use log_* and search for type:type_1, would Kibana be smart enough to skip all other indexes that don't contain type1, or does it have to actually search through all indexes under log_*?

I did a quick test on Search and Visualization and didn't see any big difference in response time between log_* and log_type_1-*, but since our logs are growing quickly, any performance tips help alot.

2

As far as I know if you run a log_* it will query all indexes even if you filter on type_1 inside the query. The indexes which are not type_1 will return 0 counts but they certainly will be queried which might add some overhead to your query timings...

3

Yeah, I'm just wondering how much overhead is added since using log_* has multiple benefits

  • Simple index pattern management
  • Simple field formatter compared to having to change field format for tens of index patterns
  • Aggregation across multiple log types

Search Profiler in Dev Tools shows quite a lot of overhead, but actually Search or Visualization between log_* and log_type_1-* shows little to no overhead. I hope someone at Elastic can confirm this.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.