# Performance degrade while querying in multiple indices **URL:** \ **Category:** Elasticsearch\ **Created:** [July 3, 2013, 8:13am UTC](https://discuss.elastic.co/t/performance-degrade-while-querying-in-multiple-indices/12635 "2013-07-03T08:13:53Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Ankit\_Jain](https://avatars.discourse-cdn.com/v4/letter/a/96bed5/32.png) [@Ankit\_Jain](https://discuss.elastic.co/u/Ankit_Jain)\ **Post date:** [July 3, 2013, 8:13am UTC](https://discuss.elastic.co/t/performance-degrade-while-querying-in-multiple-indices/12635/1 "2013-07-03T08:13:53Z")
Hi All, We are creating indices on hour basis and we are firing queries on 2 day data. The number of records expected to hit is less than 100. But, the queries are taking unpredictable time (1 sec to 1 min). Is queries will execute in all 48 indices in parallel or sequential? What will be the main reason for performance hit? Regards, Ankit Jain -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 3, 2013, 8:28am UTC](https://discuss.elastic.co/t/performance-degrade-while-querying-in-multiple-indices/12635/2 "2013-07-03T08:28:11Z")
parallel. Let's say you are using default settings, so you have 5 shards per index. You are running queries on 240 shards (that means 240 Lucene requests). Why do you need to split your data every hours? Do you see anything in logs? GC thing for example... -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 3 juil. 2013 à 10:13, Ankit Jain [ankitjaincs06@gmail.com](mailto:ankitjaincs06@gmail.com) a écrit : > Hi All, > > We are creating indices on hour basis and we are firing queries on 2 day data. The number of records expected to hit is less than 100. But, the queries are taking unpredictable time (1 sec to 1 min). > > Is queries will execute in all 48 indices in parallel or sequential? > What will be the main reason for performance hit? > > Regards, > Ankit Jain > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ankit\_Jain](https://avatars.discourse-cdn.com/v4/letter/a/96bed5/32.png) [@Ankit\_Jain](https://discuss.elastic.co/u/Ankit_Jain)\ **Post date:** [July 3, 2013, 9:27am UTC](https://discuss.elastic.co/t/performance-degrade-while-querying-in-multiple-indices/12635/3 "2013-07-03T09:27:34Z")
Hi All, Why do you need to split your data every hours? We are getting 10 million records each hour and also we would like to provide time based searching. Also, I am using below code to delete records. ``` DeleteByQueryRequestBuilder dbqb = ``` client.prepareDeleteByQuery( existingIndices.toArray(new String[existingIndices .size()])).setQuery(qb); log.info("Request done :"); DeleteByQueryResponse deleteByQueryResponse = dbqb.execute() .actionGet(); dbqb.execute().actionGet(); -\> This statement takes 1 min to get the response from ES. Regards, Ankit On Wednesday, 3 July 2013 13:58:11 UTC+5:30, David Pilato wrote: > parallel. > > Let's say you are using default settings, so you have 5 shards per index. > You are running queries on 240 shards (that means 240 Lucene requests). > > Why do you need to split your data every hours? > Do you see anything in logs? GC thing for example... > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > Le 3 juil. 2013 à 10:13, Ankit Jain \<[ankitj...@gmail.com](mailto:ankitj...@gmail.com) \\> a > écrit : > > Hi All, > > We are creating indices on hour basis and we are firing queries on 2 day > data. The number of records expected to hit is less than 100. But, the > queries are taking unpredictable time (1 sec to 1 min). > > Is queries will execute in all 48 indices in parallel or sequential? > What will be the main reason for performance hit? > > Regards, > Ankit Jain > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 3, 2013, 9:54am UTC](https://discuss.elastic.co/t/performance-degrade-while-querying-in-multiple-indices/12635/4 "2013-07-03T09:54:22Z")
You should remove the index itself. Why removing records? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 3 juil. 2013 à 11:27, Ankit Jain [ankitjaincs06@gmail.com](mailto:ankitjaincs06@gmail.com) a écrit : > Hi All, > > Why do you need to split your data every hours? > We are getting 10 million records each hour and also we would like to provide time based searching. > > Also, I am using below code to delete records. > > ``` > DeleteByQueryRequestBuilder dbqb = client.prepareDeleteByQuery( > existingIndices.toArray(new String[existingIndices > .size()])).setQuery(qb); > log.info("Request done :"); > DeleteByQueryResponse deleteByQueryResponse = dbqb.execute() > .actionGet(); > > ``` > > dbqb.execute().actionGet(); -\> This statement takes 1 min to get the response from ES. > > Regards, > Ankit > > On Wednesday, 3 July 2013 13:58:11 UTC+5:30, David Pilato wrote: > parallel. > > Let's say you are using default settings, so you have 5 shards per index. > You are running queries on 240 shards (that means 240 Lucene requests). > > Why do you need to split your data every hours? > Do you see anything in logs? GC thing for example... > > -- > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) > @dadoonet | @elasticsearchfr | @scrutmydocs > > Le 3 juil. 2013 à 10:13, Ankit Jain [ankitj...@gmail.com](mailto:ankitj...@gmail.com) a écrit : > > > Hi All, > > > > We are creating indices on hour basis and we are firing queries on 2 day data. The number of records expected to hit is less than 100. But, the queries are taking unpredictable time (1 sec to 1 min). > > > > Is queries will execute in all 48 indices in parallel or sequential? > > What will be the main reason for performance hit? > > > > Regards, > > Ankit Jain > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ankit\_Jain](https://avatars.discourse-cdn.com/v4/letter/a/96bed5/32.png) [@Ankit\_Jain](https://discuss.elastic.co/u/Ankit_Jain)\ **Post date:** [July 3, 2013, 1:35pm UTC](https://discuss.elastic.co/t/performance-degrade-while-querying-in-multiple-indices/12635/5 "2013-07-03T13:35:18Z")
Thanks for the input. On Wednesday, 3 July 2013 15:24:22 UTC+5:30, David Pilato wrote: > You should remove the index itself. Why removing records? > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > Le 3 juil. 2013 à 11:27, Ankit Jain \<[ankitj...@gmail.com](mailto:ankitj...@gmail.com) \\> a > écrit : > > Hi All, > > Why do you need to split your data every hours? > We are getting 10 million records each hour and also we would like to > provide time based searching. > > Also, I am using below code to delete records. > > ``` > DeleteByQueryRequestBuilder dbqb = > > ``` > > client.prepareDeleteByQuery( > existingIndices.toArray(new String[existingIndices > .size()])).setQuery(qb); > log.info("Request done :"); > DeleteByQueryResponse deleteByQueryResponse = > dbqb.execute() > .actionGet(); > > dbqb.execute().actionGet(); -\> This statement takes 1 min to get the > response from ES. > > Regards, > Ankit > > On Wednesday, 3 July 2013 13:58:11 UTC+5:30, David Pilato wrote: > > > parallel. > > > > Let's say you are using default settings, so you have 5 shards per index. > > You are running queries on 240 shards (that means 240 Lucene requests). > > > > Why do you need to split your data every hours? > > Do you see anything in logs? GC thing for example... > > > > -- > > _David Pilato_ | _Technical Advocate_ | \*[Elasticsearch.com](http://Elasticsearch.com)[http://elasticsearch.com/](http://elasticsearch.com/) > > \* > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > > > Le 3 juil. 2013 à 10:13, Ankit Jain [ankitj...@gmail.com](mailto:ankitj...@gmail.com) a écrit : > > > > Hi All, > > > > We are creating indices on hour basis and we are firing queries on 2 day > > data. The number of records expected to hit is less than 100. But, the > > queries are taking unpredictable time (1 sec to 1 min). > > > > Is queries will execute in all 48 indices in parallel or sequential? > > What will be the main reason for performance hit? > > > > Regards, > > Ankit Jain > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:28am UTC](https://discuss.elastic.co/t/performance-degrade-while-querying-in-multiple-indices/12635/6 "2017-07-06T02:28:25Z")