# Erally stuck while running against already existing cluster **URL:** \ **Category:** Elasticsearch\ **Tags:** rally\ **Created:** [November 5, 2020, 1:43pm UTC](https://discuss.elastic.co/t/erally-stuck-while-running-against-already-existing-cluster/254415 "2020-11-05T13:43:09Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![juriskr](https://avatars.discourse-cdn.com/v4/letter/j/51bf81/32.png) [@juriskr](https://discuss.elastic.co/u/juriskr)\ **Post date:** [November 5, 2020, 1:43pm UTC](https://discuss.elastic.co/t/erally-stuck-while-running-against-already-existing-cluster/254415/1 "2020-11-05T13:43:09Z")
I'm trying to start erally using docker based installation and the following command: docker run elastic/rally --track=nyc\_taxis --test-mode --pipeline=benchmark-only --target-hosts=, got the following output and then erally stuck with the following messages: ```auto ________ / __\____ _/ / /_ __ / /_/ / __ `/ / / / / / / _, _/ /_/ / / / /_/ / /_/ |_|\ __,_/_/_/\__ , / / ____ / [INFO] Downloading data for track nyc_taxis (30.6 kB total size) [100.0%] [INFO] Decompressing track data from [/rally/.rally/benchmarks/data/nyc_taxis/documents-1k.json.bz2] to [/rally/.rally/benchmarks/data/nyc_taxis/documents-1k.json] ... [OK] [INFO] Preparing file offset table for [/rally/.rally/benchmarks/data/nyc_taxis/documents-1k.json] ... [WARNING] merges_total_time is 11448170359 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [WARNING] merges_total_throttled_time is 9180486646 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [WARNING] indexing_total_time is 2726607864 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [WARNING] refresh_total_time is 325159125 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [WARNING] flush_total_time is 31291493 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [OK] Running delete-index [100% done] Running create-index [100% done] Running check-cluster-health [100% done] Running index [100% done] Running refresh-after-index [100% done] Running force-merge [100% done] Running refresh-after-force-merge [100% done] Running wait-until-merges-finish [0% done] ``` Can anybody please help me to understand what is wrong ? Thanks in advance. ---
**Author:** ![baz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/baz/32/5183_2.png) [@baz](https://discuss.elastic.co/u/baz)\ **Post date:** [November 5, 2020, 3:51pm UTC](https://discuss.elastic.co/t/erally-stuck-while-running-against-already-existing-cluster/254415/2 "2020-11-05T15:51:33Z")
Rally is running a [force merge](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html) operation, and waiting for it to finish. Force merges take all of your segments in your index and shrink them down to one big segment. This can take a long time if your cluster is small, or if your track is quite large. As you can see from the `io` chart in [our nightly benchmark for nyc\_taxis](https://elasticsearch-benchmarks.elastic.co/index.html#tracks/nyc-taxis/nightly/default/30d), the index is 250ish GB, which will definitely take some time. If you think its taking way too long, it might be time to look at your existing Elasticsearch cluster, to see what is going on there. The [Rally codebase](https://github.com/elastic/rally/blob/master/esrally/driver/runner.py#L675) shows what we execute to see if the Elasticsearch cluster is still waiting on the force merge to complete. You should also look in your logs to see if there are any errors that might have caused the force merge to fail but have tasks stay around. ---
**Author:** ![juriskr](https://avatars.discourse-cdn.com/v4/letter/j/51bf81/32.png) [@juriskr](https://discuss.elastic.co/u/juriskr)\ **Post date:** [November 5, 2020, 8:56pm UTC](https://discuss.elastic.co/t/erally-stuck-while-running-against-already-existing-cluster/254415/3 "2020-11-05T20:56:00Z")
Strange, becasue we have forcemerge tasks executed by curator, but they definitely not running while I was trying to benchmark with rally. I've checked this using Tasks API: ```auto curl -s 192.168.56.94:9200/_tasks?pretty | grep -i forcemerge ``` and get nothing. Cluster is green, nyc\_taxis gets created: ```auto curl -s 192.168.56.94:9200/_cat/indices | grep nyc green open nyc_taxis gt9K36KFQEaepRtx1MFZyA 1 0 1000 0 182.6kb 182.6kb ``` Don't see any issues on cluster masters logs as well as on the node, where index have been created except for this message about index creation: ```auto [2020-11-05T20:43:13,220][INFO][o.e.c.m.MetaDataCreateIndexService] [rix3-elkm1-dr] [nyc_taxis] creating index, cause [api], templates [], shards [1]/[0], mappings [type] [2020-11-05T20:43:14,728][INFO][o.e.c.r.a.AllocationService] [rix3-elkm1-dr] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[nyc_taxis][0]] ...]). ``` By the way don't know if it have any impact, but I run test using docker images of erally software and using coordinator node to access cluster. ---
**Author:** ![juriskr](https://avatars.discourse-cdn.com/v4/letter/j/51bf81/32.png) [@juriskr](https://discuss.elastic.co/u/juriskr)\ **Post date:** [November 5, 2020, 9:08pm UTC](https://discuss.elastic.co/t/erally-stuck-while-running-against-already-existing-cluster/254415/5 "2020-11-05T21:08:54Z")
By the way I was able to start esrally when I've used non-docker installation - simple pip. ---
**Author:** ![baz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/baz/32/5183_2.png) [@baz](https://discuss.elastic.co/u/baz)\ **Post date:** [November 5, 2020, 9:20pm UTC](https://discuss.elastic.co/t/erally-stuck-while-running-against-already-existing-cluster/254415/6 "2020-11-05T21:20:23Z")
ok, if you do not have any force merge tasks, its probably worth looking what the rally logs are saying about the cluster during this time period. If you can do a clean run of rally and attach the logs, we can inspect them and see what might be the issue. Also, what version of rally are you running? ---
**Author:** ![juriskr](https://avatars.discourse-cdn.com/v4/letter/j/51bf81/32.png) [@juriskr](https://discuss.elastic.co/u/juriskr)\ **Post date:** [November 6, 2020, 12:25pm UTC](https://discuss.elastic.co/t/erally-stuck-while-running-against-already-existing-cluster/254415/7 "2020-11-06T12:25:23Z")
The one I had problem with was a docker image with latest tag: ```auto $ esrally ________ / __\____ _/ / /_ __ / /_/ / __ `/ / / / / / / _, _/ /_/ / / / /_/ / /_/ |_|\ __,_/_/_/\__ , / / ____ / [ERROR] Cannot race. Only the [benchmark-only] pipeline is supported by the Rally Docker image. Add --pipeline=benchmark-only in your Rally arguments and try again. For more details read the docs for the benchmark-only pipeline in https://esrally.readthedocs.io/en/2.0.2/pipelines.html#benchmark-only Getting further help: ********************* * Check the log files in /rally/.rally/logs for errors. * Read the documentation at https://esrally.readthedocs.io/en/2.0.2/. * Ask a question on the forum at https://discuss.elastic.co/tags/c/elastic-stack/elasticsearch/rally. * Raise an issue at https://github.com/elastic/rally/issues and include the log files in /rally/.rally/logs. ------------------------------- [INFO] FAILURE (took 4 seconds) ------------------------------- $ esrally --version esrally 2.0.2 $ ``` Also I don't have the same issue with the esrally installed using pip. Will try to run esrally using dockre a bit later and be back with logs. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 4, 2020, 12:25pm UTC](https://discuss.elastic.co/t/erally-stuck-while-running-against-already-existing-cluster/254415/8 "2020-12-04T12:25:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.