# Esrally gets stuck on check-cluster-health

**URL:** <https://discuss.elastic.co/t/esrally-gets-stuck-on-check-cluster-health/291264>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [December 8, 2021, 9:09pm UTC](https://discuss.elastic.co/t/esrally-gets-stuck-on-check-cluster-health/291264 "2021-12-08T21:09:47Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![vtex](https://avatars.discourse-cdn.com/v4/letter/v/b5a626/32.png) [@vtex](https://discuss.elastic.co/u/vtex)\
**Post date:** [December 8, 2021, 9:09pm UTC](https://discuss.elastic.co/t/esrally-gets-stuck-on-check-cluster-health/291264/1 "2021-12-08T21:09:48Z")

</div>

Hi,

I'm running a local/single-node esrally. My esrally version is 2.30. When I run the below command the execution hangs while checking the status of the cluster -- i.e. in check-cluster-health.

$ esrally race --distribution-version=7.16.0 --track=nyc\_taxis --challenge=append-no-conflicts

Checking the health gives a green result:  
$ curl [http://localhost:39200/\_cat/health?v](http://localhost:39200/_cat/health?v)  
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending\_tasks max\_task\_wait\_time active\_shards\_percent  
1638996487 20:48:07 rally-benchmark green 1 1 2 2 0 0 0 0 - 100.0%

But, what I see in the rally.log is:  
2021-12-08 20:47:15,330 -not-actor-/PID:92289 Elasticsearch WARNING GET [http://127.0.0.1:39200/\_cluster/he](http://127.0.0.1:39200/_cluster/he) alth/nyc\_taxis?wait\_for\_status=green&wait\_for\_no\_relocating\_shards=false [status:408 request:30.003s]

Other similar questions couldn't fix the issue for me.

Thanks for the help.

---

<div class="post-metadata">

**Author:** ![json](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/json/32/4125_2.png) [@json](https://discuss.elastic.co/u/json)\
**Post date:** [December 8, 2021, 10:38pm UTC](https://discuss.elastic.co/t/esrally-gets-stuck-on-check-cluster-health/291264/2 "2021-12-08T22:38:19Z")

</div>

Hi Alpha,

Would you mind running your `curl` check using the same endpoint as Rally? I.e.,

```auto
curl 'http://127.0.0.1:39200/_cluster/health/nyc_taxis?wait_for_status=green&wait_for_no_relocating_shards=false'

```

Also, make sure there are no other process leftovers from other `esrally` executions.

Can you share more of `rally.log`, preferably the complete file and any customized configurations?

`GET /_cluster/health/<index>` can behave this way for non-existing indices or if the index will never be green. `_cat/health` returns `green` because it represents the status of all known indices in the cluster.

Some possibilities:

- The `nyc_taxis` index exists but is `yellow`. This will happen if the number of index replicas has been configured to be \>0 for a single node cluster.
- `nyc_taxis` does not exist and is, therefore `red`.

See also [Rally race gets stuck on check-cluster-health](https://discuss.elastic.co/t/rally-race-gets-stuck-on-check-cluster-health/268815).

---

<div class="post-metadata">

**Author:** ![vtex](https://avatars.discourse-cdn.com/v4/letter/v/b5a626/32.png) [@vtex](https://discuss.elastic.co/u/vtex)\
**Post date:** [December 9, 2021, 12:18am UTC](https://discuss.elastic.co/t/esrally-gets-stuck-on-check-cluster-health/291264/3 "2021-12-09T00:18:34Z")

</div>

```auto
$ curl -X GET http://127.0.0.1:39200/_cluster/health/nyc_taxis
{"cluster_name":"rally-benchmark","status":"red","timed_out":true,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

```

The first time I run esrally, there is no issue. But when I try to re-run again (after a clean completion of the first run), the issue comes. Currently, I'm using the default configuration and not changing anything. That is, to repeat the problem, what I have to do is run it twice.

```auto
$ esrally race --distribution-version=7.16.0 --track=nyc_taxis --challenge=append-no-conflicts
... completes successfully.
$ esrally race --distribution-version=7.16.0 --track=nyc_taxis --challenge=append-no-conflicts
... waits forever for check-cluster-health

```

---

<div class="post-metadata">

**Author:** ![Quentin\_Pradet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quentin_pradet/32/94192_2.png) [@Quentin\_Pradet](https://discuss.elastic.co/u/Quentin_Pradet)\
**Post date:** [December 14, 2021, 6:32am UTC](https://discuss.elastic.co/t/esrally-gets-stuck-on-check-cluster-health/291264/4 "2021-12-14T06:32:58Z")

</div>

Since running the whole nyc\_taxis challenge is slow on my laptop, I added `--test-mode` to your commands, and I fail to reproduce the issue. Do you also have the issue when you add `--test-mode`?

Also, what do you call "the first run"? What do you need to reset to get into "first run" conditions?

Is there a reason why you could not share your Rally configuration and logs, as requested by Jason?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 11, 2022, 6:33am UTC](https://discuss.elastic.co/t/esrally-gets-stuck-on-check-cluster-health/291264/5 "2022-01-11T06:33:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.