Failure running Rally under OSX

Elastic Stack Elasticsearch
1

After successfully installing Rally 0.5.3 under OS X 10.11.6, and invoking the following command:

esrally --track=pmc --target-hosts=localhost:9200 --pipeline=benchmark-only

I am seeing the following errors consistently:

[ERROR] Could not initialize successfully.

[ERROR] Are there are still processes from a previous race?
[ERROR] Please check and terminate related Python processes before running Rally again.

[ERROR] Cannot race. ActorAddr-(T|:1900) is not a valid ActorSystem admin.

There are no stuck Rally processes running. I have tried repeating right after a fresh reboot, no help.

Has anyone else run into this issue?

Here is the related log output:

2017-06-19 20:50:46,566 rally.main INFO OS [posix.uname_result(sysname='Darwin', nodename='bherring-osx.local', release='15.6.0', version='Darwin Kernel Version 15.6.0: Tue Apr 11 16:00:51 PDT 2017; root:xnu-3248.60.11.5.3~1/RELEASE_X86_64', machine='x86_64')]
2017-06-19 20:50:46,566 rally.main INFO Python [namespace(cache_tag='cpython-34', hexversion=50595056, name='cpython', version=sys.version_info(major=3, minor=4, micro=4, releaselevel='final', serial=0))]
2017-06-19 20:50:46,566 rally.main INFO Rally version [0.5.3]
2017-06-19 20:50:46,566 rally.main INFO Command line arguments: Namespace(advanced_config=False, assume_defaults=False, auto_manage_indices=None, car='defaults', challenge=None, client_options='timeout:60000,request_timeout:60000', cluster_health='green', configuration_name=None, data_paths=None, distribution_repository='release', distribution_version='', effective_start_date=datetime.datetime(2017, 6, 19, 20, 50, 46, 563299), enable_driver_profiling=False, laps=1, logging='file', offline=False, override_src_dir=None, pipeline='benchmark-only', preserve_install='False', quiet=False, report_file='', report_format='markdown', revision='current', subcommand=None, target_hosts='ts-brian-edg-EsELB-YII28AP49D82-976867935.us-east-1.elb.amazonaws.com:80', telemetry='', test_mode=False, track='pmc', track_repository='default', user_tag='')
2017-06-19 20:50:46,566 rally.net INFO Rally connects directly to the Internet (no proxy support).
2017-06-19 20:50:49,489 rally.main INFO Detected a working Internet connection.
2017-06-19 20:50:49,505 rally.process INFO Skipping myself (PID [2158]).
2017-06-19 20:50:49,506 rally.main INFO Actor system already running locally? [False]
2017-06-19 20:50:49,506 rally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'Convention Address.IPv4': '127.0.0.1:1900', 'ip': '127.0.0.1'}].
2017-06-19 20:50:50,163 root INFO ++++ Actor System gen (3, 6) started, admin @ ActorAddr-(T|:1900)
2017-06-19 20:50:55,158 rally.actor ERROR Could not initialize internal actor system. Terminating.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/esrally/actor.py", line 193, in bootstrap_actor_system
capabilities=capabilities)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/thespian/actors.py", line 619, in init
systemBase, capabilities, logDefs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/thespian/actors.py", line 657, in _startupActorSys
systemBase = sbc(self, logDefs=logDefs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/thespian/system/multiprocTCPBase.py", line 27, in init
super(ActorSystemBase, self).init(system, logDefs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/thespian/system/multiprocCommon.py", line 84, in init
super(multiprocessCommon, self).init(system, logDefs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/thespian/system/systemBase.py", line 87, in init
logDefs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/thespian/system/multiprocCommon.py", line 111, in _startAdmin
raise InvalidActorAddress(adminAddr, 'not a valid ActorSystem admin')
thespian.actors.InvalidActorAddress: ActorAddr-(T|:1900) is not a valid ActorSystem admin
2017-06-19 20:50:55,164 root ERROR A fatal error occurred while running subcommand [race].
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/esrally/rally.py", line 429, in dispatch_sub_command
race(cfg)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/esrally/rally.py", line 375, in race
with_actor_system(lambda c: racecontrol.run(c), cfg)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/esrally/rally.py", line 382, in with_actor_system
actors = actor.bootstrap_actor_system(try_join=already_running, prefer_local_only=not already_running)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/esrally/actor.py", line 193, in bootstrap_actor_system
capabilities=capabilities)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/thespian/actors.py", line 619, in init
systemBase, capabilities, logDefs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/thespian/actors.py", line 657, in _startupActorSys
systemBase = sbc(self, logDefs=logDefs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/thespian/system/multiprocTCPBase.py", line 27, in init
super(ActorSystemBase, self).init(system, logDefs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/thespian/system/multiprocCommon.py", line 84, in init
super(multiprocessCommon, self).init(system, logDefs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/thespian/system/systemBase.py", line 87, in init
logDefs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/thespian/system/multiprocCommon.py", line 111, in _startAdmin
raise InvalidActorAddress(adminAddr, 'not a valid ActorSystem admin')
thespian.actors.InvalidActorAddress: ActorAddr-(T|:1900) is not a valid ActorSystem admin

2

Hi @bherring,

based on what I see in your logs - and especially the following lines, your problem is not related to any hanging processes.

Rally starts a so-called actor system internally to let its internal components communicate with each other. This actor system has one process that listens on port 1900. You can see from the logs that it has started it successfully but then failed further down the road. You can also see that there is a 5 second delay between the last two log messages. You run into a timeout upon startup of the actor system. This could maybe be caused by a firewall that is running on your system. So you should check your security settings and also whether you run a third-party firewall on your Mac.

You can also run the following command in a separate terminal window before you start Rally:

lsof -P -n -r 1 -i :1900

It shows activity on port 1900 on a high level once every second. On my machine it looks something like this:

daniel@io:~ $ lsof -P -n -r 1 -i :1900
=======
=======
=======
=======
=======
=======
COMMAND  PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
Python  1560 daniel   11u  IPv4 0xa4062a472885ea3b      0t0  TCP *:1900 (LISTEN)
Python  1560 daniel   13u  IPv4 0xa4062a4728910333      0t0  TCP 192.168.2.15:1900->192.168.2.15:50092 (ESTABLISHED)
Python  1561 daniel   11u  IPv4 0xa4062a472885ea3b      0t0  TCP *:1900 (LISTEN)
Python  1561 daniel   17u  IPv4 0xa4062a4728dac143      0t0  TCP 192.168.2.15:50092->192.168.2.15:1900 (ESTABLISHED)
=======
COMMAND  PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
Python  1560 daniel   11u  IPv4 0xa4062a472885ea3b      0t0  TCP *:1900 (LISTEN)
Python  1560 daniel   13u  IPv4 0xa4062a4728910333      0t0  TCP 192.168.2.15:1900->192.168.2.15:50092 (ESTABLISHED)
Python  1561 daniel   11u  IPv4 0xa4062a472885ea3b      0t0  TCP *:1900 (LISTEN)
Python  1561 daniel   17u  IPv4 0xa4062a4728dac143      0t0  TCP 192.168.2.15:50092->192.168.2.15:1900 (ESTABLISHED)

I hope that helps you to resolve the problem.

Daniel

3

Thanks Daniel! I was able to workaround this issue by running Rally from a different machine without the firewall issue.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.