# Rally gets stuck on check-cluster-health **URL:** \ **Category:** Elasticsearch\ **Tags:** rally\ **Created:** [January 28, 2021, 10:44am UTC](https://discuss.elastic.co/t/rally-gets-stuck-on-check-cluster-health/262480 "2021-01-28T10:44:51Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![rsarpal](https://avatars.discourse-cdn.com/v4/letter/r/ac91a4/32.png) [@rsarpal](https://discuss.elastic.co/u/rsarpal)\ **Post date:** [January 28, 2021, 10:44am UTC](https://discuss.elastic.co/t/rally-gets-stuck-on-check-cluster-health/262480/1 "2021-01-28T10:44:51Z")
Hi, We have older version of elastic 2.4 , I am trying to test it with rally. I have tried to test with latest and the older versions of rally (1.3.0, 1.4.1, latest) Rally gets stuck on check-cluster-health Running put-settings [100% done] Running delete-index [100% done] Running create-index [100% done] Running check-cluster-health [0% done] Rally log shows : ```auto 2021-01-27 10:06:41,94 -not-actor-/PID:50 elasticsearch WARNING GET http://x.x.x.x:9200/_cluster/health/queries?wait_ for_status=green&wait_for_no_relocating_shards=true [status:408 request:30.004s] ``` ES 2.4 throws following error : ```auto [2021-01-28 10:27:27,544][WARN][rest.suppressed] path: /pmc, params: {index=pmc} java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Map at org.elasticsearch.action.admin.indices.create.CreateIndexRequest.source(CreateIndexRequest.java:393) at org.elasticsearch.action.admin.indices.create.CreateIndexRequest.source(CreateIndexRequest.java:368) at org.elasticsearch.rest.action.admin.indices.create.RestCreateIndexAction.handleRequest(RestCreateIndexAction.java:47) at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54) at org.elasticsearch.rest.RestController.executeHandler(RestController.java:198) at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:158) at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:153) at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:101) at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:451) at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:61) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60) at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [January 28, 2021, 1:35pm UTC](https://discuss.elastic.co/t/rally-gets-stuck-on-check-cluster-health/262480/2 "2021-01-28T13:35:47Z")
Rally dropped 2.x support since [https://github.com/elastic/rally/pull/1105](https://github.com/elastic/rally/pull/1105) (note ES 2.x is EOL since February 28 2018). Additionally the corresponding 2 and 1 branches from rally-tracks (you seem to be attempting to use `pmc`) have been also removed around that date. However the tag [v2](https://github.com/elastic/rally-tracks/releases/tag/v2) is still present, so you could use a version of Rally prior to the [afforementioned commit](https://github.com/elastic/rally/pull/1105) (Rally \<2.0.3) with something like: `./rally --track-revision=v2 --pipeline=benchmark-only --target-hosts=127.0.0.1:9200 --track=geonames --challenge=append-no-conflicts-index-only --kill-running-processes` ---
**Author:** ![rsarpal](https://avatars.discourse-cdn.com/v4/letter/r/ac91a4/32.png) [@rsarpal](https://discuss.elastic.co/u/rsarpal)\ **Post date:** [January 29, 2021, 10:56am UTC](https://discuss.elastic.co/t/rally-gets-stuck-on-check-cluster-health/262480/3 "2021-01-29T10:56:02Z")
Great ! this helped and it is working locally for me now. However, the kubernetes cluster where i want to run doesn't have internet. 1. So locally i have checkout to v2 branch for tracks. 2. And then i am downloading percolator track data as a tar, extracting it in .rally folder. [rally-tracks/download.sh at master · elastic/rally-tracks · GitHub](https://github.com/elastic/rally-tracks/blob/master/download.sh) 3. building a docker image using downloaded .rally folder, and pushing it on an artifactory. 4. and then running the artifactory image in cluster But now its giving this error. ```auto rally@rally:~$ esrally race --track=percolator --test-mode --pipeline=benchmark-only --target-hosts=10.9.202.6 --kill-running -processes --track-revision=v2 ________ / __\____ _/ / /_ __ / /_/ / __ `/ / / / / / / _, _/ /_/ / / / /_/ / /_/ |_|\ __,_/_/_/\__ , / / ____ / [WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled. [ERROR] Cannot race. Error in track preparator (('Could not register track plugin at [/rally/.rally/benchmarks/tracks/default/percolator]', None)) ``` ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [January 29, 2021, 5:15pm UTC](https://discuss.elastic.co/t/rally-gets-stuck-on-check-cluster-health/262480/4 "2021-01-29T17:15:13Z")
It's getting rather complex in a non internet connect env, but you'd basically need to download the rally track / percolator using the `v2` tag (not master) and then have rally directly reference that directory using [--track-path](https://esrally.readthedocs.io/en/stable/command_line_reference.html#track-path) ---
**Author:** ![rsarpal](https://avatars.discourse-cdn.com/v4/letter/r/ac91a4/32.png) [@rsarpal](https://discuss.elastic.co/u/rsarpal)\ **Post date:** [February 1, 2021, 8:56am UTC](https://discuss.elastic.co/t/rally-gets-stuck-on-check-cluster-health/262480/5 "2021-02-01T08:56:39Z")
I downloaded the v2 tracks under .rally/benchmarks/track/v2 and tried to test it on the container using: ```auto esrally race --test-mode --pipeline=benchmark-only --track-path=~/.rally/benchmarks/tracks/v2/percolator/track .json --target-hosts=x.x.x.x --kill-running-processes ``` But it throws the error ```auto [WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled. [ERROR] Cannot race. Error in track preparator (('Could not register track plugin at [/rally/.rally/benchmarks/tracks/v2/percolator]', None)) ``` log file: ```auto 2021-02-01 08:49:35,302 -not-actor-/PID:3066 esrally.rally ERROR Cannot run subcommand [race]. Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/esrally/rally.py", line 770, in dispatch_sub_command race(cfg) File "/usr/local/lib/python3.8/site-packages/esrally/rally.py", line 680, in race with_actor_system(racecontrol.run, cfg) File "/usr/local/lib/python3.8/site-packages/esrally/rally.py", line 708, in with_actor_system runnable(cfg) File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 375, in run raise e File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 372, in run pipeline(cfg) File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 60, in __call__ self.target(cfg) File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 305, in benchmark_only return race(cfg, external=True) File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 251, in race raise exceptions.RallyError(result.message, result.cause) esrally.exceptions.RallyError: ("Error in track preparator (('Could not register track plugin at [/rally/.rally/benchmarks/tracks/v2/percolator]', None))", None) ``` ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [February 3, 2021, 7:47am UTC](https://discuss.elastic.co/t/rally-gets-stuck-on-check-cluster-health/262480/6 "2021-02-03T07:47:22Z")
You should use an older version of Rally with the v2 tag of rally-tracks. I successfully used the latest Rally 1.4 ([esrally · PyPI](https://pypi.org/project/esrally/1.4.1/)) against ES 2.x referencing a v2 checkout of rally-tracks using: `./rally --pipeline=benchmark-only --track-path= --challenge=append-no-conflicts --target-hosts=127.0.0.1:9200 --on-error=abort` ---
**Author:** ![rsarpal](https://avatars.discourse-cdn.com/v4/letter/r/ac91a4/32.png) [@rsarpal](https://discuss.elastic.co/u/rsarpal)\ **Post date:** [February 4, 2021, 8:04am UTC](https://discuss.elastic.co/t/rally-gets-stuck-on-check-cluster-health/262480/7 "2021-02-04T08:04:10Z")
Finally it worked with this for me esrally --track=percolator --test-mode --pipeline=benchmark-only --target-hosts=ip:9200 --offline --distribution-version=2.0.0 --track-revision=v2 Thanks for your help and suggestions 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 4, 2021, 8:04am UTC](https://discuss.elastic.co/t/rally-gets-stuck-on-check-cluster-health/262480/8 "2021-03-04T08:04:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.