Error running rally on existing cluster

Hi ,

I am following http://esrally.readthedocs.io/en/stable/recipes.html#benchmarking-an-existing-cluster docs and trying to run rally on existing elastic search and getting below error

esrally --track=pmc --target-hosts=IPAddress:9300 --pipeline=benchmark-only

Error - [ERROR] Cannot race. ('Could not execute benchmark', ConnectionError('N/A', "('Connection aborted.', BadStatusLine('This is not a HTTP port',))", ProtocolError('Connection aborted.', BadStatusLine('This is not a HTTP port',))))

I tried using client-options as well but didn't work.

Rally is implemented in Python and therefore uses the HTTP client, so you need to use port 9200.

1 Like

Thanks . I will try with port 9200 and will post results here

I have two more questions :
Docs mentioned that we can define multiple target host ..e.g. ```
--target-hosts=10.5.5.10:9200,10.5.5.11:9200,10.5.5.12:9200

1) Do we need to define all data nodes or master nodes here in target host or both ?
2) If Suppose , an ELB is being used then what should be command to run rally?

@Alp1 in general, please open a new topic when you have a new question. Otherwise, other users that hit a similar problem might have a hard time finding out that there is already a solution somewhere on the forum deep down in another discussion.

Do we need to define all data nodes or master nodes here in target host or both ?

You should add all nodes here that you want to receive traffic. This is usually your data nodes but not the master-eligible nodes.

If Suppose , an ELB is being used then what should be command to run rally?

I'm not familiar enough with ELB but I suppose you'd then only target the ELB.

2 Likes

Thanks .. I will test with all data nodes and will post results here

We are using some other port as http port not 9200 . I tried with other port now the error message is changed. Is it mandatory to use 9200 ?

Here are the logs -

2017-12-14 15:39:11,41 PID:28252 rally.main INFO Detected a working Internet connection.

2017-12-14 15:39:11,57 PID:28252 rally.process INFO Skipping myself (PID [28252]).

2017-12-14 15:39:11,57 PID:28252 rally.main INFO Actor system already running locally? [False]

2017-12-14 15:39:11,57 PID:28252 rally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'coordinator': True}].

2017-12-14 15:39:11,75 PID:28258 root INFO ++++ Actor System gen (3, 8) started, admin @ ActorAddr-(T|:1900)

2017-12-14 15:39:11,84 PID:28252 rally.racecontrol INFO User specified pipeline [benchmark-only].

2017-12-14 15:39:11,85 PID:28252 rally.racecontrol INFO Using configured hosts [{'port': 8089, 'host': '10.205.1.130'}]

2017-12-14 15:39:11,85 PID:28252 rally.actor INFO Joining already running actor system with system base [multiprocTCPBase].

2017-12-14 15:39:11,86 PID:28258 rally.actor INFO Capabilities [{'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'coordinator': True, 'Thespian Generation': (3, 8), 'Thespian Version': '1513265951070', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian Watch Supported': True, 'Thespian ActorSystem Version': 2, 'Python Version': (3, 4, 3, 'final', 0)}] match requirements [{'coordinator': True}].

2017-12-14 15:39:11,86 PID:28258 rally.actor INFO Capabilities [{'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'coordinator': True, 'Thespian Generation': (3, 8), 'Thespian Version': '1513265951070', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian Watch Supported': True, 'Thespian ActorSystem Version': 2, 'Python Version': (3, 4, 3, 'final', 0)}] match requirements [{'coordinator': True}].

2017-12-14 15:39:11,93 PID:28260 rally.racecontrol INFO BenchmarkActor#receiveMessage(msg = [<class 'esrally.racecontrol.Setup'>] sender = [ActorAddr-(T|:57182)])

2017-12-14 15:39:11,93 PID:28260 rally.actor INFO Capabilities [{'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'coordinator': True, 'Thespian Generation': (3, 8), 'Thespian Version': '1513265951070', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian Watch Supported': True, 'Thespian ActorSystem Version': 2, 'Python Version': (3, 4, 3, 'final', 0)}] match requirements [{'coordinator': True}].

2017-12-14 15:39:11,94 PID:28260 rally.client INFO Creating ES client connected to [{'port': 8089, 'host': 'IPAddress'}] with options [{'basic_auth_password': 'changeme', 'basic_auth_user': 'elastic'}]

2017-12-14 15:39:11,98 PID:28260 rally.racecontrol INFO Automatically derived distribution version [5.6.0]

2017-12-14 15:39:11,317 PID:28260 rally.repo INFO Checking out [5] in [/home/bhs515/.rally/benchmarks/tracks/default] for distribution version [5.6.0].

2017-12-14 15:39:11,322 PID:28260 rally.racecontrol ERROR BenchmarkActor encountered a fatal exception. Shutting down.

Traceback (most recent call last):

File "/usr/local/lib/python3.4/dist-packages/esrally/utils/repo.py", line 45, in update

git.checkout(self.repo_dir, branch=branch)

File "/usr/local/lib/python3.4/dist-packages/esrally/utils/git.py", line 18, in probe

return f(src, *args, **kwargs)

File "/usr/local/lib/python3.4/dist-packages/esrally/utils/git.py", line 50, in checkout

raise exceptions.SupplyError("Could not checkout branch [%s]. Do you have uncommitted changes?" % branch)

esrally.exceptions.SupplyError: ('Could not checkout branch [5]. Do you have uncommitted changes?', None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.4/dist-packages/esrally/racecontrol.py", line 100, in receiveMessage

self.setup(msg)

File "/usr/local/lib/python3.4/dist-packages/esrally/racecontrol.py", line 188, in setup

t = track.load_track(self.cfg)

File "/usr/local/lib/python3.4/dist-packages/esrally/track/loader.py", line 63, in load_track

repo = track_repo(cfg)

File "/usr/local/lib/python3.4/dist-packages/esrally/track/loader.py", line 124, in track_repo

return GitTrackRepository(cfg, fetch, update)

File "/usr/local/lib/python3.4/dist-packages/esrally/track/loader.py", line 152, in init

self.repo.update(distribution_version)

File "/usr/local/lib/python3.4/dist-packages/esrally/utils/repo.py", line 67, in update

raise exceptions.DataError("Cannot update %s in [%s] (%s)." % (self.resource_name, self.repo_dir, e.message)).with_traceback(tb)

File "/usr/local/lib/python3.4/dist-packages/esrally/utils/repo.py", line 45, in update

git.checkout(self.repo_dir, branch=branch)

File "/usr/local/lib/python3.4/dist-packages/esrally/utils/git.py", line 18, in probe

return f(src, *args, **kwargs)

File "/usr/local/lib/python3.4/dist-packages/esrally/utils/git.py", line 50, in checkout

raise exceptions.SupplyError("Could not checkout branch [%s]. Do you have uncommitted changes?" % branch)

esrally.exceptions.DataError: ('Cannot update tracks in [/home/user/.rally/benchmarks/tracks/default] (Could not checkout branch [5]. Do you have uncommitted changes?).', None)

2017-12-14 15:39:11,324 PID:28252 rally.racecontrol ERROR A benchmark failure has occurred

2017-12-14 15:39:11,324 PID:28252 rally.racecontrol INFO Telling benchmark actor to exit.

2017-12-14 15:39:11,325 PID:28252 rally.main INFO Attempting to shutdown internal actor system.

2017-12-14 15:39:11,325 PID:28260 rally.racecontrol INFO BenchmarkActor#receiveMessage(msg = [<class 'thespian.actors.ActorExitRequest'>] sender = [ActorAddr-(T|:57182)])

2017-12-14 15:39:11,326 PID:28261 rally.mechanic INFO MechanicActor#receiveMessage(msg = [<class 'thespian.actors.ActorExitRequest'>] sender = [ActorAddr-(T|:27460)])

2017-12-14 15:39:11,326 PID:28261 rally.actor INFO Transitioning from [None] to [cluster_stopping].

2017-12-14 15:39:11,328 PID:28260 rally.racecontrol INFO BenchmarkActor#receiveMessage(msg = [<class 'thespian.actors.ActorExitRequest'>] sender = [ActorAddr-(T|:1900)])

2017-12-14 15:39:11,329 PID:28260 rally.racecontrol INFO BenchmarkActor#receiveMessage(msg = [<class 'thespian.actors.ChildActorExited'>] sender = [ActorAddr-(T|:27287)])

2017-12-14 15:39:11,329 PID:28260 rally.racecontrol INFO BenchmarkActor received unknown message [ChildActorExited:ActorAddr-(T|:27287)] (ignoring).

2017-12-14 15:39:11,334 PID:28258 root INFO ---- Actor System shutdown

2017-12-14 15:39:11,335 PID:28252 rally.main INFO Actor system is still running. Waiting...

No, port 9200 is not mandatory, you can use an arbitrary port.

The problem seems to be that you have modified files in /home/user/.rally/benchmarks/tracks/default. I guess that you have edited the files and then ran the benchmark against Elasticsearch 5.6.0. In that case, Rally tries to switch to a different version of the tracks in /home/user/.rally/benchmarks/tracks/default by checking out the branch 5. And as you have local modifications (on the master branch) it cannot do that and will fail.

I suggest that you do either of the following:

  • Do your changes on the correct git branch
  • Revert your changes (e.g. by just deleting /home/user/.rally/benchmarks/tracks/default. Rally will take care of cloning the repo again for you).

If you want to create your own benchmarks, it is better to just start with a simple json file and not adapt the standard tracks (see the tutorial in the docs on how to do that).

can't down the data?

@danielmitterdorfer Perfect. Thanks a lot .
I had created a file during R&D. I removed the file and it worked :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.