Hello everyone, I was running a benchmark-only for a cluster and ran into an error as described below. I would really appreciate it if someone can help me figure out why is this error occurring and a possible resolution.
I am using pyenv to manage my version of python, I used python 3.6.8 and finally 3.8.2 after noticing the requirement on the esrally install page. Got the same error in both cases.
I was running the following command -
esrally --track-path=<absolute path> --target-hosts=<URL> --pipeline=benchmark-only --client-options="basic_auth_user:'',basic_auth_password:'',verify_certs:false"
I ran the command twice, the second time after it had failed so the errors after 2020-07-10 15:24:05,567 would be related to the time the command was ran the second time.
These are the resulting logs -
2020-07-10 15:23:41,535 -not-actor-/PID:51873 esrally.rally INFO Race id [aaaaa-bbbb-cccc-dddd-eeeeeeee]
2020-07-10 15:23:41,535 -not-actor-/PID:51873 esrally.rally INFO OS [uname_result(system='Darwin', node='xxxxyyyy', release='19.3.0', version='Darwin Kernel Version 19.3.0: Thu Jan 9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64', machine='x86_64', processor='i386')]
2020-07-10 15:23:41,535 -not-actor-/PID:51873 esrally.rally INFO Python [namespace(_multiarch='darwin', cache_tag='cpython-38', hexversion=50856688, name='cpython', version=sys.version_info(major=3, minor=8, micro=2, releaselevel='final', serial=0))]
2020-07-10 15:23:41,535 -not-actor-/PID:51873 esrally.rally INFO Rally version [2.0.0]
2020-07-10 15:23:41,536 -not-actor-/PID:51873 esrally.utils.net INFO Connecting directly to the Internet (no proxy support).
2020-07-10 15:23:42,234 -not-actor-/PID:51873 esrally.rally INFO Detected a working Internet connection.
2020-07-10 15:23:42,301 -not-actor-/PID:51873 esrally.rally INFO Actor system already running locally? [False]
2020-07-10 15:23:42,301 -not-actor-/PID:51873 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': 'xxxx', 'Convention Address.IPv4': 'xxxx'}].
2020-07-10 15:23:42,388 -not-actor-/PID:51873 root WARNING Unable to get address info for address xxxxyyyy (AddressFamily.AF_INET, SocketKind.SOCK_DGRAM, 17, 0): <class 'socket.gaierror'> [Errno 8] nodename nor servname provided, or not known
2020-07-10 15:23:42,390 -not-actor-/PID:51873 root WARNING Unable to get address info for address xxxxyyyy (AddressFamily.AF_INET, SocketKind.SOCK_DGRAM, 17, 0): <class 'socket.gaierror'> [Errno 8] nodename nor servname provided, or not known
2020-07-10 15:23:42,392 -not-actor-/PID:51873 root WARNING Unable to get address info for address xxxxyyyy (AddressFamily.AF_INET, SocketKind.SOCK_DGRAM, 17, AddressInfo.AI_PASSIVE): <class 'socket.gaierror'> [Errno 8] nodename nor servname provided, or not known
2020-07-10 15:23:42,394 -not-actor-/PID:51873 root WARNING Unable to get address info for address xxxxyyyy (AddressFamily.AF_INET, SocketKind.SOCK_DGRAM, 17, AddressInfo.AI_PASSIVE): <class 'socket.gaierror'> [Errno 8] nodename nor servname provided, or not known
2020-07-10 15:23:42,410 -not-actor-/PID:51877 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)
2020-07-10 15:23:47,406 -not-actor-/PID:51873 esrally.actor ERROR Could not initialize internal actor system.
Traceback (most recent call last):
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/actor.py", line 248, in bootstrap_actor_system
return thespian.actors.ActorSystem(system_base,
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/actors.py", line 635, in __init__
systemBase = self._startupActorSys(
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/actors.py", line 676, in _startupActorSys
systemBase = sbc(self, logDefs=logDefs)
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/system/multiprocTCPBase.py", line 28, in __init__
super(ActorSystemBase, self).__init__(system, logDefs)
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/system/multiprocCommon.py", line 86, in __init__
super(multiprocessCommon, self).__init__(system, logDefs)
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/system/systemBase.py", line 326, in __init__
self._startAdmin(self.adminAddr,
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/system/multiprocCommon.py", line 114, in _startAdmin
raise InvalidActorAddress(adminAddr,
thespian.actors.InvalidActorAddress: ActorAddr-(T|:1900) is not a valid ActorSystem admin
2020-07-10 15:23:47,414 -not-actor-/PID:51873 esrally.rally INFO Falling back to offline actor system.
2020-07-10 15:23:47,415 -not-actor-/PID:51873 esrally.actor INFO Joining already running actor system with system base [multiprocQueueBase].
2020-07-10 15:23:47,438 -not-actor-/PID:51880 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-Q.ThespianQ
2020-07-10 15:23:47,460 -not-actor-/PID:51873 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2020-07-10 15:23:47,460 -not-actor-/PID:51873 esrally.racecontrol INFO Using configured hosts [{'host': 'xxxx', 'port': xxxx, 'use_ssl': True}]
2020-07-10 15:23:47,461 -not-actor-/PID:51873 esrally.actor INFO Joining already running actor system with system base [multiprocQueueBase].
2020-07-10 15:23:50,474 -not-actor-/PID:51873 esrally.rally INFO Attempting to shutdown internal actor system.
2020-07-10 15:23:50,500 -not-actor-/PID:51880 root INFO ---- Actor System shutdown
2020-07-10 15:23:50,502 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:51,506 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:52,512 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:53,517 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:54,521 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:55,524 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:56,529 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:57,533 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:58,537 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:59,541 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:24:00,547 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:24:01,552 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:24:02,556 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:24:03,561 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:24:04,564 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:24:05,567 -not-actor-/PID:51873 esrally.rally WARNING Shutdown timed out. Actor system is still running.
2020-07-10 15:24:05,568 -not-actor-/PID:51873 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 362, in run
pipeline(cfg)
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 60, in __call__
self.target(cfg)
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 298, in benchmark_only
return race(cfg, external=True)
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 241, in race
benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True})
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/actors.py", line 702, in createActor
return self._systemBase.newPrimaryActor(actorClass,
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/system/systemBase.py", line 205, in newPrimaryActor
raise NoCompatibleSystemForActor(
thespian.actors.NoCompatibleSystemForActor: No compatible ActorSystem could be found for Actor <class 'esrally.racecontrol.BenchmarkActor'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/rally.py", line 766, in dispatch_sub_command
race(cfg)
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/rally.py", line 676, in race
with_actor_system(racecontrol.run, cfg)
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/rally.py", line 704, in with_actor_system
runnable(cfg)
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 370, in run
raise exceptions.RallyError("This race ended with a fatal crash.").with_traceback(tb)
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 362, in run
pipeline(cfg)
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 60, in __call__
self.target(cfg)
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 298, in benchmark_only
return race(cfg, external=True)
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 241, in race
benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True})
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/actors.py", line 702, in createActor
return self._systemBase.newPrimaryActor(actorClass,
File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/system/systemBase.py", line 205, in newPrimaryActor
raise NoCompatibleSystemForActor(
esrally.exceptions.RallyError: ('This race ended with a fatal crash.', None)
I am very new to this forum, please excuse me if I missed something or my format is messed up. Suggestions are always welcome.