Rally gets stuck on check-cluster-health

Elastic Stack Elasticsearch
1

Hi,

We have older version of elastic 2.4 , I am trying to test it with rally. I have tried to test with latest and the older versions of rally (1.3.0, 1.4.1, latest)

Rally gets stuck on check-cluster-health

Running put-settings [100% done]
Running delete-index [100% done]
Running create-index [100% done]
Running check-cluster-health [ 0% done]

Rally log shows :

2021-01-27 10:06:41,94 -not-actor-/PID:50 elasticsearch WARNING GET http://x.x.x.x:9200/_cluster/health/queries?wait_
for_status=green&wait_for_no_relocating_shards=true [status:408 request:30.004s]

ES 2.4 throws following error :

[2021-01-28 10:27:27,544][WARN ][rest.suppressed          ] path: /pmc, params: {index=pmc}
java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Map
        at org.elasticsearch.action.admin.indices.create.CreateIndexRequest.source(CreateIndexRequest.java:393)
        at org.elasticsearch.action.admin.indices.create.CreateIndexRequest.source(CreateIndexRequest.java:368)
        at org.elasticsearch.rest.action.admin.indices.create.RestCreateIndexAction.handleRequest(RestCreateIndexAction.java:47)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54)
        at org.elasticsearch.rest.RestController.executeHandler(RestController.java:198)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:158)
        at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:153)
        at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:101)
        at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:451)
        at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:61)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2

Rally dropped 2.x support since https://github.com/elastic/rally/pull/1105 (note ES 2.x is EOL since February 28 2018).

Additionally the corresponding 2 and 1 branches from rally-tracks (you seem to be attempting to use pmc) have been also removed around that date. However the tag v2 is still present, so you could use a version of Rally prior to the afforementioned commit (Rally <2.0.3) with something like:

./rally --track-revision=v2 --pipeline=benchmark-only --target-hosts=127.0.0.1:9200 --track=geonames --challenge=append-no-conflicts-index-only --kill-running-processes

2 Likes
3

Great ! this helped and it is working locally for me now.

However, the kubernetes cluster where i want to run doesn't have internet.

  1. So locally i have checkout to v2 branch for tracks.

  2. And then i am downloading percolator track data as a tar, extracting it in .rally folder.
    rally-tracks/download.sh at master · elastic/rally-tracks · GitHub

  3. building a docker image using downloaded .rally folder, and pushing it on an artifactory.

  4. and then running the artifactory image in cluster

But now its giving this error.

rally@rally:~$ esrally race --track=percolator --test-mode --pipeline=benchmark-only --target-hosts=10.9.202.6 --kill-running
-processes --track-revision=v2

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled.

[ERROR] Cannot race. Error in track preparator (('Could not register track plugin at [/rally/.rally/benchmarks/tracks/default/percolator]', None))
4

It's getting rather complex in a non internet connect env, but you'd basically need to download the rally track / percolator using the v2 tag (not master) and then have rally directly reference that directory using --track-path

5

I downloaded the v2 tracks under .rally/benchmarks/track/v2 and tried to test it on the container using:

esrally race --test-mode --pipeline=benchmark-only --track-path=~/.rally/benchmarks/tracks/v2/percolator/track
.json --target-hosts=x.x.x.x --kill-running-processes

But it throws the error

[WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled.

[ERROR] Cannot race. Error in track preparator (('Could not register track plugin at [/rally/.rally/benchmarks/tracks/v2/percolator]', None))

log file:

2021-02-01 08:49:35,302 -not-actor-/PID:3066 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/esrally/rally.py", line 770, in dispatch_sub_command
    race(cfg)
  File "/usr/local/lib/python3.8/site-packages/esrally/rally.py", line 680, in race
    with_actor_system(racecontrol.run, cfg)
  File "/usr/local/lib/python3.8/site-packages/esrally/rally.py", line 708, in with_actor_system
    runnable(cfg)
  File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 375, in run
    raise e
  File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 372, in run
    pipeline(cfg)
  File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 60, in __call__
    self.target(cfg)
  File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 305, in benchmark_only
    return race(cfg, external=True)
  File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 251, in race
    raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: ("Error in track preparator (('Could not register track plugin at [/rally/.rally/benchmarks/tracks/v2/percolator]', None))", None)
6

You should use an older version of Rally with the v2 tag of rally-tracks.

I successfully used the latest Rally 1.4 (esrally · PyPI) against ES 2.x referencing a v2 checkout of rally-tracks using:

./rally --pipeline=benchmark-only --track-path=<local_check_out_to_v2_tag_of_rally_tracks/percolator> --challenge=append-no-conflicts --target-hosts=127.0.0.1:9200 --on-error=abort

1 Like
7

Finally it worked with this for me

esrally --track=percolator --test-mode --pipeline=benchmark-only --target-hosts=ip:9200 --offline --distribution-version=2.0.0 --track-revision=v2

Thanks for your help and suggestions :slight_smile:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.