Error running rally on existing cluster

Alp1 · December 12, 2017, 3:58pm

Hi ,

I am following http://esrally.readthedocs.io/en/stable/recipes.html#benchmarking-an-existing-cluster docs and trying to run rally on existing elastic search and getting below error

esrally --track=pmc --target-hosts=IPAddress:9300 --pipeline=benchmark-only

Error - [ERROR] Cannot race. ('Could not execute benchmark', ConnectionError('N/A', "('Connection aborted.', BadStatusLine('This is not a HTTP port',))", ProtocolError('Connection aborted.', BadStatusLine('This is not a HTTP port',))))

I tried using client-options as well but didn't work.

Christian_Dahlqvist · December 12, 2017, 4:00pm

Rally is implemented in Python and therefore uses the HTTP client, so you need to use port 9200.

Alp1 · December 12, 2017, 4:03pm

Thanks . I will try with port 9200 and will post results here

Alp1 · December 12, 2017, 5:24pm

I have two more questions :
Docs mentioned that we can define multiple target host ..e.g. ```
--target-hosts=10.5.5.10:9200,10.5.5.11:9200,10.5.5.12:9200

1) Do we need to define all data nodes or master nodes here in target host or both ?
2) If Suppose , an ELB is being used then what should be command to run rally?

danielmitterdorfer · December 13, 2017, 7:30am

@Alp1 in general, please open a new topic when you have a new question. Otherwise, other users that hit a similar problem might have a hard time finding out that there is already a solution somewhere on the forum deep down in another discussion.

Do we need to define all data nodes or master nodes here in target host or both ?

You should add all nodes here that you want to receive traffic. This is usually your data nodes but not the master-eligible nodes.

If Suppose , an ELB is being used then what should be command to run rally?

I'm not familiar enough with ELB but I suppose you'd then only target the ELB.

Alp1 · December 13, 2017, 4:20pm

Thanks .. I will test with all data nodes and will post results here

Alp1 · December 14, 2017, 4:03pm

We are using some other port as http port not 9200 . I tried with other port now the error message is changed. Is it mandatory to use 9200 ?

Here are the logs -

2017-12-14 15:39:11,41 PID:28252 rally.main INFO Detected a working Internet connection.

2017-12-14 15:39:11,57 PID:28252 rally.process INFO Skipping myself (PID [28252]).

2017-12-14 15:39:11,57 PID:28252 rally.main INFO Actor system already running locally? [False]

2017-12-14 15:39:11,57 PID:28252 rally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'coordinator': True}].

2017-12-14 15:39:11,75 PID:28258 root INFO ++++ Actor System gen (3, 8) started, admin @ ActorAddr-(T|:1900)

2017-12-14 15:39:11,84 PID:28252 rally.racecontrol INFO User specified pipeline [benchmark-only].

2017-12-14 15:39:11,85 PID:28252 rally.racecontrol INFO Using configured hosts [{'port': 8089, 'host': '10.205.1.130'}]

2017-12-14 15:39:11,85 PID:28252 rally.actor INFO Joining already running actor system with system base [multiprocTCPBase].

2017-12-14 15:39:11,86 PID:28258 rally.actor INFO Capabilities [{'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'coordinator': True, 'Thespian Generation': (3, 8), 'Thespian Version': '1513265951070', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian Watch Supported': True, 'Thespian ActorSystem Version': 2, 'Python Version': (3, 4, 3, 'final', 0)}] match requirements [{'coordinator': True}].

2017-12-14 15:39:11,93 PID:28260 rally.racecontrol INFO BenchmarkActor#receiveMessage(msg = [<class 'esrally.racecontrol.Setup'>] sender = [ActorAddr-(T|:57182)])

2017-12-14 15:39:11,93 PID:28260 rally.actor INFO Capabilities [{'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'coordinator': True, 'Thespian Generation': (3, 8), 'Thespian Version': '1513265951070', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian Watch Supported': True, 'Thespian ActorSystem Version': 2, 'Python Version': (3, 4, 3, 'final', 0)}] match requirements [{'coordinator': True}].

2017-12-14 15:39:11,94 PID:28260 rally.client INFO Creating ES client connected to [{'port': 8089, 'host': 'IPAddress'}] with options [{'basic_auth_password': 'changeme', 'basic_auth_user': 'elastic'}]

2017-12-14 15:39:11,98 PID:28260 rally.racecontrol INFO Automatically derived distribution version [5.6.0]

2017-12-14 15:39:11,317 PID:28260 rally.repo INFO Checking out [5] in [/home/bhs515/.rally/benchmarks/tracks/default] for distribution version [5.6.0].

2017-12-14 15:39:11,322 PID:28260 rally.racecontrol ERROR BenchmarkActor encountered a fatal exception. Shutting down.

Traceback (most recent call last):

File "/usr/local/lib/python3.4/dist-packages/esrally/utils/repo.py", line 45, in update

git.checkout(self.repo_dir, branch=branch)

File "/usr/local/lib/python3.4/dist-packages/esrally/utils/git.py", line 18, in probe

return f(src, *args, **kwargs)

File "/usr/local/lib/python3.4/dist-packages/esrally/utils/git.py", line 50, in checkout

raise exceptions.SupplyError("Could not checkout branch [%s]. Do you have uncommitted changes?" % branch)

esrally.exceptions.SupplyError: ('Could not checkout branch [5]. Do you have uncommitted changes?', None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.4/dist-packages/esrally/racecontrol.py", line 100, in receiveMessage

self.setup(msg)

File "/usr/local/lib/python3.4/dist-packages/esrally/racecontrol.py", line 188, in setup

t = track.load_track(self.cfg)

File "/usr/local/lib/python3.4/dist-packages/esrally/track/loader.py", line 63, in load_track

repo = track_repo(cfg)

File "/usr/local/lib/python3.4/dist-packages/esrally/track/loader.py", line 124, in track_repo

return GitTrackRepository(cfg, fetch, update)

File "/usr/local/lib/python3.4/dist-packages/esrally/track/loader.py", line 152, in init

self.repo.update(distribution_version)

File "/usr/local/lib/python3.4/dist-packages/esrally/utils/repo.py", line 67, in update

raise exceptions.DataError("Cannot update %s in [%s] (%s)." % (self.resource_name, self.repo_dir, e.message)).with_traceback(tb)

File "/usr/local/lib/python3.4/dist-packages/esrally/utils/repo.py", line 45, in update

git.checkout(self.repo_dir, branch=branch)

File "/usr/local/lib/python3.4/dist-packages/esrally/utils/git.py", line 18, in probe

return f(src, *args, **kwargs)

File "/usr/local/lib/python3.4/dist-packages/esrally/utils/git.py", line 50, in checkout

raise exceptions.SupplyError("Could not checkout branch [%s]. Do you have uncommitted changes?" % branch)

esrally.exceptions.DataError: ('Cannot update tracks in [/home/user/.rally/benchmarks/tracks/default] (Could not checkout branch [5]. Do you have uncommitted changes?).', None)

2017-12-14 15:39:11,324 PID:28252 rally.racecontrol ERROR A benchmark failure has occurred

2017-12-14 15:39:11,324 PID:28252 rally.racecontrol INFO Telling benchmark actor to exit.

2017-12-14 15:39:11,325 PID:28252 rally.main INFO Attempting to shutdown internal actor system.

2017-12-14 15:39:11,325 PID:28260 rally.racecontrol INFO BenchmarkActor#receiveMessage(msg = [<class 'thespian.actors.ActorExitRequest'>] sender = [ActorAddr-(T|:57182)])

2017-12-14 15:39:11,326 PID:28261 rally.mechanic INFO MechanicActor#receiveMessage(msg = [<class 'thespian.actors.ActorExitRequest'>] sender = [ActorAddr-(T|:27460)])

2017-12-14 15:39:11,326 PID:28261 rally.actor INFO Transitioning from [None] to [cluster_stopping].

2017-12-14 15:39:11,328 PID:28260 rally.racecontrol INFO BenchmarkActor#receiveMessage(msg = [<class 'thespian.actors.ActorExitRequest'>] sender = [ActorAddr-(T|:1900)])

2017-12-14 15:39:11,329 PID:28260 rally.racecontrol INFO BenchmarkActor#receiveMessage(msg = [<class 'thespian.actors.ChildActorExited'>] sender = [ActorAddr-(T|:27287)])

2017-12-14 15:39:11,329 PID:28260 rally.racecontrol INFO BenchmarkActor received unknown message [ChildActorExited:ActorAddr-(T|:27287)] (ignoring).

2017-12-14 15:39:11,334 PID:28258 root INFO ---- Actor System shutdown

2017-12-14 15:39:11,335 PID:28252 rally.main INFO Actor system is still running. Waiting...

danielmitterdorfer · December 14, 2017, 4:35pm

No, port 9200 is not mandatory, you can use an arbitrary port.

The problem seems to be that you have modified files in /home/user/.rally/benchmarks/tracks/default. I guess that you have edited the files and then ran the benchmark against Elasticsearch 5.6.0. In that case, Rally tries to switch to a different version of the tracks in /home/user/.rally/benchmarks/tracks/default by checking out the branch 5. And as you have local modifications (on the master branch) it cannot do that and will fail.

I suggest that you do either of the following:

Do your changes on the correct git branch
Revert your changes (e.g. by just deleting /home/user/.rally/benchmarks/tracks/default. Rally will take care of cloning the repo again for you).

If you want to create your own benchmarks, it is better to just start with a simple json file and not adapt the standard tracks (see the tutorial in the docs on how to do that).

zqc0512 · December 15, 2017, 6:57am

can't down the data?

Alp1 · December 15, 2017, 9:08pm

@danielmitterdorfer Perfect. Thanks a lot .
I had created a file during R&D. I removed the file and it worked

system · January 12, 2018, 9:08pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.