# Esrally stuck Running large\_terms **URL:** \ **Category:** Elasticsearch\ **Tags:** rally\ **Created:** [July 25, 2018, 10:09am UTC](https://discuss.elastic.co/t/esrally-stuck-running-large-terms/141541 "2018-07-25T10:09:37Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Andrew\_Yu](https://avatars.discourse-cdn.com/v4/letter/a/85e7bf/32.png) [@Andrew\_Yu](https://discuss.elastic.co/u/Andrew_Yu)\ **Post date:** [July 25, 2018, 10:09am UTC](https://discuss.elastic.co/t/esrally-stuck-running-large-terms/141541/1 "2018-07-25T10:09:37Z")
Hi, I was running esrally to test the performance of an elasticsearch cluster of version 5.6.10. Esrally version is 1.0.0. It stucks when running large\_terms at percentage 3% for a very long time( more than 4 hours). Here's the logs: [INFO] Racing on track [geonames], challenge [append-no-conflicts] and car ['external'] with version [5.6.10]. [WARNING] merges\_total\_time is 15064528 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [WARNING] merges\_total\_throttled\_time is 114305 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [WARNING] indexing\_total\_time is 3101013 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [WARNING] refresh\_total\_time is 3659868 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [WARNING] flush\_total\_time is 15527 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. Running delete-index [100% done] Running create-index [100% done] Running check-cluster-health [100% done] Running index-append [100% done] Running refresh-after-index [100% done] Running force-merge [100% done] Running refresh-after-force-merge [100% done] Running index-stats [100% done] Running node-stats [100% done] Running default [100% done] Running term [100% done] Running phrase [100% done] Running country\_agg\_uncached [100% done] Running country\_agg\_cached [100% done] Running scroll [100% done] Running expression [100% done] Running painless\_static [100% done] Running painless\_dynamic [100% done] Running large\_terms [3% done] Any idea about this? Thanks ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [July 25, 2018, 10:50am UTC](https://discuss.elastic.co/t/esrally-stuck-running-large-terms/141541/2 "2018-07-25T10:50:47Z")
Hi, is this reproducible? Did you check the Elasticsearch logs of the cluster that you are targeting with your benchmark? Did you check the Rally logs in `~/.rally/logs/rally.log`? Daniel ---
**Author:** ![Andrew\_Yu](https://avatars.discourse-cdn.com/v4/letter/a/85e7bf/32.png) [@Andrew\_Yu](https://discuss.elastic.co/u/Andrew_Yu)\ **Post date:** [July 26, 2018, 6:36am UTC](https://discuss.elastic.co/t/esrally-stuck-running-large-terms/141541/3 "2018-07-26T06:36:36Z")
Maybe it's because I'm using the default jvm options. I've increased Xms and Xmx and try again. ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [July 26, 2018, 7:00am UTC](https://discuss.elastic.co/t/esrally-stuck-running-large-terms/141541/4 "2018-07-26T07:00:55Z")
I think I know what's going on (or rather not going on...). The `large terms` task intentionally sends a very large number of terms to Elasticsearch and you are benchmarking a 5.x cluster with the `benchmark-only` pipeline. The default limit for `indices.query.bool.max_clause_count` is 1024 but the benchmark needs a limit of 50000. As this is a static cluster setting you need to add it yourself to `elasticsearch.yml`, i.e.: ```auto indices.query.bool.max_clause_count: 50000 ``` If you have Rally setup the cluster for you, it would do it but there is no chance we can set it for you with the `benchmark-only` pipeline. I think we emit a warning in the logs but we should probably be more vocal about it and show a warning that you need to ensure you set it yourself. Can you please change `elasticsearch.yml` on each node accordingly, restart and retry the benchmark? ---
**Author:** ![Andrew\_Yu](https://avatars.discourse-cdn.com/v4/letter/a/85e7bf/32.png) [@Andrew\_Yu](https://discuss.elastic.co/u/Andrew_Yu)\ **Post date:** [July 27, 2018, 1:54am UTC](https://discuss.elastic.co/t/esrally-stuck-running-large-terms/141541/5 "2018-07-27T01:54:47Z")
I've added this parameters then tested again, and esrally completed the test successfully this time! Thank you very much 🙂 ---
**Author:** ![Andrew\_Yu](https://avatars.discourse-cdn.com/v4/letter/a/85e7bf/32.png) [@Andrew\_Yu](https://discuss.elastic.co/u/Andrew_Yu)\ **Post date:** [July 27, 2018, 1:57am UTC](https://discuss.elastic.co/t/esrally-stuck-running-large-terms/141541/6 "2018-07-27T01:57:24Z")
> [@danielmitterdorfer](#): > > max\_clause\_count I missed the comment, thanks for the remind. I'll check the logs more carefully next time. ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [July 27, 2018, 4:48am UTC](https://discuss.elastic.co/t/esrally-stuck-running-large-terms/141541/7 "2018-07-27T04:48:56Z")
Hi, thanks for your feedback and glad it is resolved now. This behavior is really trappy and we will improve it. I raised [https://github.com/elastic/rally/issues/541](https://github.com/elastic/rally/issues/541) so in the future Rally will show a clear and actionable warning message to avoid such problems. Daniel ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 24, 2018, 4:56am UTC](https://discuss.elastic.co/t/esrally-stuck-running-large-terms/141541/8 "2018-08-24T04:56:18Z")
Closed on August 24, 2018, 4:56am UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.