# Migration from single to multi-index increased searchRate and overall search time **URL:** \ **Category:** Elasticsearch\ **Created:** [June 7, 2021, 5:51am UTC](https://discuss.elastic.co/t/migration-from-single-to-multi-index-increased-searchrate-and-overall-search-time/275102 "2021-06-07T05:51:14Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![code\_merchant](https://avatars.discourse-cdn.com/v4/letter/c/b5ac83/32.png) [@code\_merchant](https://discuss.elastic.co/u/code_merchant)\ **Post date:** [June 7, 2021, 5:51am UTC](https://discuss.elastic.co/t/migration-from-single-to-multi-index-increased-searchrate-and-overall-search-time/275102/1 "2021-06-07T05:51:14Z")
Earlier, we had only 1 index of storing all the data of the cluster of around 400GB. We migrated from that architecture to a new one that has monthly indices each of size around 30 GB (recommended index size in ES). After this migration, the search queries over multiple indices using the template-alias(search-alias-\* or search-alias-read) is taking more time than the earlier architecture with one index. In the new architecture we are using 1 replica shard for each month data. we are storing data for past 2 years, so there are currently 25 indices. It was expected that new architecture will be more robust and flexible, am I missing something? From Cluster Performance Metrics Cluster 1: (One index - 400 GB) SearchLatency: 35 ms (per index/shard) SearchRate: 40 operation/min ThreadpoolSearchThreads: ~0 threads Cluster2: (Multi index - 25 indices each of 30 GB) SearchLatency: 4 ms (per index/shard) SearchRate: 500-1000 operation/min (Number of operations gets multiplied by number of shards on which searching is performed) ThreadpoolSearchThreads: ~15-17 threads ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 7, 2021, 5:54am UTC](https://discuss.elastic.co/t/migration-from-single-to-multi-index-increased-searchrate-and-overall-search-time/275102/2 "2021-06-07T05:54:16Z")
Welcome to our community! 😃 I am not sure I am following here, as your provided information suggests that latency is lower, and search rate is higher for the multi index cluster. Not the other way around. ---
**Author:** ![code\_merchant](https://avatars.discourse-cdn.com/v4/letter/c/b5ac83/32.png) [@code\_merchant](https://discuss.elastic.co/u/code_merchant)\ **Post date:** [June 7, 2021, 6:03am UTC](https://discuss.elastic.co/t/migration-from-single-to-multi-index-increased-searchrate-and-overall-search-time/275102/3 "2021-06-07T06:03:55Z")
For the same ingestion rate, When the query retrieves data from Cluster 1(single-index) it is taking less time and when it retrieves from Cluster 2(multi-index) it takes more time. The number of search operation in Cluster 2 got multiplied due to multiple indices present in Cluster 2 over which searching is perform thus increased the total search time of the query ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 7, 2021, 6:05am UTC](https://discuss.elastic.co/t/migration-from-single-to-multi-index-increased-searchrate-and-overall-search-time/275102/4 "2021-06-07T06:05:55Z")
> [@code\_merchant](#): > > When the query retrieves data from Cluster 1(single-index) it is taking less time and when it retrieves from Cluster 2(multi-index) it takes more time. That's not what you are showing in your first post. > [@code\_merchant](#): > > Cluster 1: (One index - 400 GB) > SearchLatency: 35 ms Versus > [@code\_merchant](#): > > Cluster2: (Multi index - 25 indices each of 30 GB) > SearchLatency: 4 ms As 35 \> 4. ---
**Author:** ![code\_merchant](https://avatars.discourse-cdn.com/v4/letter/c/b5ac83/32.png) [@code\_merchant](https://discuss.elastic.co/u/code_merchant)\ **Post date:** [June 7, 2021, 6:23am UTC](https://discuss.elastic.co/t/migration-from-single-to-multi-index-increased-searchrate-and-overall-search-time/275102/5 "2021-06-07T06:23:53Z")
Query is: Elastic Search Query ```auto {"from":0,"size":3000,"query":{"bool":{"must":[{"exists":{"field":"field1","boost":1.0}}],"filter":[{"term":{"field2":{"value":"170291438512","boost":1.0}}}],"must_not":[{"term":{"field3":{"value":"18977-96339-6797436","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"_source":{"includes":["field4","field5","field6","field3","field7","field8","field9","field10","field11"],"excludes":[]},"sort":[{"field3":{"order":"desc"}}]} ``` Cluster A is taking: 5-9.5 ms Cluster B is taking: 12-18 ms for this query ---
**Author:** ![code\_merchant](https://avatars.discourse-cdn.com/v4/letter/c/b5ac83/32.png) [@code\_merchant](https://discuss.elastic.co/u/code_merchant)\ **Post date:** [June 7, 2021, 6:25am UTC](https://discuss.elastic.co/t/migration-from-single-to-multi-index-increased-searchrate-and-overall-search-time/275102/6 "2021-06-07T06:25:42Z")
I am new to ES, search latency is time taken by 1 index(1 shard per index, in my case) to process the query not the whole query over all the shards? ---
**Author:** ![code\_merchant](https://avatars.discourse-cdn.com/v4/letter/c/b5ac83/32.png) [@code\_merchant](https://discuss.elastic.co/u/code_merchant)\ **Post date:** [June 7, 2021, 6:27am UTC](https://discuss.elastic.co/t/migration-from-single-to-multi-index-increased-searchrate-and-overall-search-time/275102/7 "2021-06-07T06:27:58Z")
GET index-name/\_doc/\_search in first cluster GET read-index-alias/\_doc/\_search in second cluster read-index-alias = [index-2019-05, index-2019-06, ..., index-2021-05, index-2021-06] ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2021, 6:28am UTC](https://discuss.elastic.co/t/migration-from-single-to-multi-index-increased-searchrate-and-overall-search-time/275102/8 "2021-07-05T06:28:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.