Migration from single to multi-index increased searchRate and overall search time

Elastic Stack Elasticsearch
1

Earlier, we had only 1 index of storing all the data of the cluster of around 400GB. We migrated from that architecture to a new one that has monthly indices each of size around 30 GB (recommended index size in ES). After this migration, the search queries over multiple indices using the template-alias(search-alias-* or search-alias-read) is taking more time than the earlier architecture with one index. In the new architecture we are using 1 replica shard for each month data. we are storing data for past 2 years, so there are currently 25 indices. It was expected that new architecture will be more robust and flexible, am I missing something?

From Cluster Performance Metrics
Cluster 1: (One index - 400 GB)
SearchLatency: 35 ms (per index/shard)
SearchRate: 40 operation/min
ThreadpoolSearchThreads: ~0 threads

Cluster2: (Multi index - 25 indices each of 30 GB)
SearchLatency: 4 ms (per index/shard)
SearchRate: 500-1000 operation/min (Number of operations gets multiplied by number of shards on which searching is performed)
ThreadpoolSearchThreads: ~15-17 threads

2

Welcome to our community! :smiley:

I am not sure I am following here, as your provided information suggests that latency is lower, and search rate is higher for the multi index cluster. Not the other way around.

3

For the same ingestion rate,
When the query retrieves data from Cluster 1(single-index) it is taking less time and when it retrieves from Cluster 2(multi-index) it takes more time. The number of search operation in Cluster 2 got multiplied due to multiple indices present in Cluster 2 over which searching is perform thus increased the total search time of the query

4

That's not what you are showing in your first post.

Versus

As 35 > 4.

5

Query is:
Elastic Search Query

{"from":0,"size":3000,"query":{"bool":{"must":[{"exists":{"field":"field1","boost":1.0}}],"filter":[{"term":{"field2":{"value":"170291438512","boost":1.0}}}],"must_not":[{"term":{"field3":{"value":"18977-96339-6797436","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"_source":{"includes":["field4","field5","field6","field3","field7","field8","field9","field10","field11"],"excludes":[]},"sort":[{"field3":{"order":"desc"}}]}

Cluster A is taking: 5-9.5 ms
Cluster B is taking: 12-18 ms for this query

6

I am new to ES, search latency is time taken by 1 index(1 shard per index, in my case) to process the query not the whole query over all the shards?

7

GET index-name/_doc/_search in first cluster

GET read-index-alias/_doc/_search in second cluster

read-index-alias = [index-2019-05, index-2019-06, ..., index-2021-05, index-2021-06]

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.