[Error] Unable to run esrally benchmark in local

gmittal22 · May 23, 2024, 8:34am

Command used:
esrally race --distribution-version=7.10.0 --track=geonames --challenge=append-no-conflicts --target-hosts=127.0.0.1:9200 --kill-running-processes --pipeline=benchmark-only

Logs:

2024-05-23 04:51:49,486 -not-actor-/PID:63923 esrally.rally INFO Actor system already running locally? [False]
2024-05-23 04:51:49,486 -not-actor-/PID:63923 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].
2024-05-23 04:51:49,509 -not-actor-/PID:63927 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)
2024-05-23 04:51:54,512 -not-actor-/PID:63923 esrally.actor ERROR Could not initialize internal actor system.
Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/actor.py", line 263, in bootstrap_actor_system
    return thespian.actors.ActorSystem(system_base, logDefs=log.load_configuration(), capabilities=capabilities)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 635, in __init__
    systemBase = self._startupActorSys(
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 676, in _startupActorSys
    systemBase = sbc(self, logDefs=logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocTCPBase.py", line 28, in __init__
    super(ActorSystemBase, self).__init__(system, logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocCommon.py", line 86, in __init__
    super(multiprocessCommon, self).__init__(system, logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/systemBase.py", line 326, in __init__
    self._startAdmin(self.adminAddr,
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocCommon.py", line 114, in _startAdmin
    raise InvalidActorAddress(adminAddr,
thespian.actors.InvalidActorAddress: ActorAddr-(T|:1900) is not a valid ActorSystem admin
2024-05-23 04:51:54,531 -not-actor-/PID:63923 esrally.rally INFO Falling back to offline actor system.
2024-05-23 04:51:54,557 -not-actor-/PID:63947 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-Q.ThespianQ
2024-05-23 04:51:54,565 -not-actor-/PID:63923 esrally.racecontrol INFO Race id is [8a8357a2-a6a0-4c9b-b00e-338a2957b19f]
2024-05-23 04:51:54,565 -not-actor-/PID:63923 esrally.racecontrol INFO User specified no pipeline. Automatically derived pipeline [from-distribution].
2024-05-23 04:51:54,566 -not-actor-/PID:63923 esrally.racecontrol INFO Setting default host to [127.0.0.1:39200]
2024-05-23 04:51:57,579 -not-actor-/PID:63923 esrally.rally INFO Attempting to shutdown internal actor system.
2024-05-23 04:51:57,606 -not-actor-/PID:63947 root INFO ---- Actor System shutdown
2024-05-23 04:51:57,608 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:51:58,611 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:51:59,614 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:52:00,616 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:52:01,619 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:52:02,621 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:52:03,627 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:52:04,631 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:52:05,634 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:52:06,640 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:52:07,642 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:52:08,648 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:52:09,650 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:52:10,657 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:52:11,660 -not-actor-/PID:63923 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 04:52:12,661 -not-actor-/PID:63923 esrally.rally WARNING Shutdown timed out. Actor system is still running.
2024-05-23 04:52:12,662 -not-actor-/PID:63923 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 396, in run
    pipeline(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 72, in __call__
    self.target(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 328, in from_distribution
    return race(cfg, distribution=True)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 283, in race
    benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True})
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 702, in createActor
    return self._systemBase.newPrimaryActor(actorClass,
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/systemBase.py", line 205, in newPrimaryActor
    raise NoCompatibleSystemForActor(
thespian.actors.NoCompatibleSystemForActor: No compatible ActorSystem could be found for Actor <class 'esrally.racecontrol.BenchmarkActor'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 1183, in dispatch_sub_command
    race(cfg, args.kill_running_processes)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 931, in race
    with_actor_system(racecontrol.run, cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 961, in with_actor_system
    runnable(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 405, in run
    raise exceptions.RallyError("This race ended with a fatal crash.").with_traceback(tb)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 396, in run
    pipeline(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 72, in __call__
    self.target(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 328, in from_distribution
    return race(cfg, distribution=True)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 283, in race
    benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True})
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 702, in createActor
    return self._systemBase.newPrimaryActor(actorClass,
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/systemBase.py", line 205, in newPrimaryActor
    raise NoCompatibleSystemForActor(
esrally.exceptions.RallyError: This race ended with a fatal crash.
2024-05-23 05:22:10,316 -not-actor-/PID:72522 esrally.rally INFO OS [uname_result(system='Darwin', node='Gunits-MacBook-Air.local', release='23.4.0', version='Darwin Kernel Version 23.4.0: Fri Mar 15 00:19:22 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T8112', machine='arm64')]
2024-05-23 05:22:10,317 -not-actor-/PID:72522 esrally.rally INFO Python [namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=6, releaselevel='final', serial=0), hexversion=50923248, _multiarch='darwin')]
2024-05-23 05:22:10,317 -not-actor-/PID:72522 esrally.rally INFO Rally version [2.10.0]
2024-05-23 05:22:10,317 -not-actor-/PID:72522 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-05-23 05:22:10,317 -not-actor-/PID:72522 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-05-23 05:22:10,317 -not-actor-/PID:72522 esrally.rally INFO Cleaning track dependency directory [/Users/gunitmittal/.rally/libs]...
2024-05-23 05:22:10,336 -not-actor-/PID:72522 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 1183, in dispatch_sub_command
    race(cfg, args.kill_running_processes)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 929, in race
    raise exceptions.RallyError(msg)
esrally.exceptions.RallyError: There are other Rally processes running on this machine (PIDs: [63927, 63928]) but only one Rally benchmark is allowed to run at the same time.

You can use --kill-running-processes flag to kill running processes automatically and allow Rally to continue to run a new benchmark. Otherwise, you need to manually kill them.
2024-05-23 05:29:36,853 -not-actor-/PID:74292 esrally.rally INFO OS [uname_result(system='Darwin', node='Gunits-MacBook-Air.local', release='23.4.0', version='Darwin Kernel Version 23.4.0: Fri Mar 15 00:19:22 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T8112', machine='arm64')]
2024-05-23 05:29:36,854 -not-actor-/PID:74292 esrally.rally INFO Python [namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=6, releaselevel='final', serial=0), hexversion=50923248, _multiarch='darwin')]
2024-05-23 05:29:36,854 -not-actor-/PID:74292 esrally.rally INFO Rally version [2.10.0]
2024-05-23 05:29:36,854 -not-actor-/PID:74292 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-05-23 05:29:36,854 -not-actor-/PID:74292 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-05-23 05:29:36,854 -not-actor-/PID:74292 esrally.rally INFO Cleaning track dependency directory [/Users/gunitmittal/.rally/libs]...
2024-05-23 05:29:36,854 -not-actor-/PID:74292 esrally.rally INFO Killing running Rally processes
2024-05-23 05:29:36,868 -not-actor-/PID:74292 esrally.utils.process INFO Killing lingering process with PID [63927] and command line [['/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python', '/Users/gunitmittal/Desktop/rally/.venv/bin/esrally', 'race', '--distribution-version=6.8.0', '--track=geonames']].
2024-05-23 05:29:37,872 -not-actor-/PID:74292 esrally.utils.process INFO Killing lingering process with PID [63928] and command line [['/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python', '/Users/gunitmittal/Desktop/rally/.venv/bin/esrally', 'race', '--distribution-version=6.8.0', '--track=geonames']].
2024-05-23 05:29:38,891 -not-actor-/PID:74292 esrally.rally INFO Actor system already running locally? [False]
2024-05-23 05:29:38,891 -not-actor-/PID:74292 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].
2024-05-23 05:29:38,920 -not-actor-/PID:74304 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)
2024-05-23 05:29:43,920 -not-actor-/PID:74292 esrally.actor ERROR Could not initialize internal actor system.
Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/actor.py", line 263, in bootstrap_actor_system
    return thespian.actors.ActorSystem(system_base, logDefs=log.load_configuration(), capabilities=capabilities)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 635, in __init__
    systemBase = self._startupActorSys(
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 676, in _startupActorSys
    systemBase = sbc(self, logDefs=logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocTCPBase.py", line 28, in __init__
    super(ActorSystemBase, self).__init__(system, logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocCommon.py", line 86, in __init__
    super(multiprocessCommon, self).__init__(system, logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/systemBase.py", line 326, in __init__
    self._startAdmin(self.adminAddr,
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocCommon.py", line 114, in _startAdmin
    raise InvalidActorAddress(adminAddr,
thespian.actors.InvalidActorAddress: ActorAddr-(T|:1900) is not a valid ActorSystem admin
2024-05-23 05:29:43,941 -not-actor-/PID:74292 esrally.rally INFO Falling back to offline actor system.
2024-05-23 05:29:43,956 -not-actor-/PID:74334 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-Q.ThespianQ
2024-05-23 05:29:43,968 -not-actor-/PID:74292 esrally.racecontrol INFO Race id is [c6f10af6-245b-4e2f-ab7b-f4101086cf8d]
2024-05-23 05:29:43,968 -not-actor-/PID:74292 esrally.racecontrol INFO User specified no pipeline. Automatically derived pipeline [from-distribution].
2024-05-23 05:29:43,969 -not-actor-/PID:74292 esrally.racecontrol INFO Setting default host to [127.0.0.1:39200]
2024-05-23 05:29:46,980 -not-actor-/PID:74292 esrally.rally INFO Attempting to shutdown internal actor system.
2024-05-23 05:29:47,8 -not-actor-/PID:74334 root INFO ---- Actor System shutdown
2024-05-23 05:29:47,12 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:29:48,14 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:29:49,19 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:29:50,23 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:29:51,28 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:29:52,31 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:29:53,35 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:29:54,40 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:29:55,44 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:29:56,46 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:29:57,52 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:29:58,58 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:29:59,60 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:30:00,66 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:30:01,68 -not-actor-/PID:74292 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:30:02,74 -not-actor-/PID:74292 esrally.rally WARNING Shutdown timed out. Actor system is still running.
2024-05-23 05:30:02,75 -not-actor-/PID:74292 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 396, in run
    pipeline(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 72, in __call__
    self.target(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 328, in from_distribution
    return race(cfg, distribution=True)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 283, in race
    benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True})
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 702, in createActor
    return self._systemBase.newPrimaryActor(actorClass,
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/systemBase.py", line 205, in newPrimaryActor
    raise NoCompatibleSystemForActor(
thespian.actors.NoCompatibleSystemForActor: No compatible ActorSystem could be found for Actor <class 'esrally.racecontrol.BenchmarkActor'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 1183, in dispatch_sub_command
    race(cfg, args.kill_running_processes)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 931, in race
    with_actor_system(racecontrol.run, cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 961, in with_actor_system
    runnable(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 405, in run
    raise exceptions.RallyError("This race ended with a fatal crash.").with_traceback(tb)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 396, in run
    pipeline(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 72, in __call__
    self.target(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 328, in from_distribution
    return race(cfg, distribution=True)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 283, in race
    benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True})
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 702, in createActor
    return self._systemBase.newPrimaryActor(actorClass,
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/systemBase.py", line 205, in newPrimaryActor
    raise NoCompatibleSystemForActor(
esrally.exceptions.RallyError: This race ended with a fatal crash.
2024-05-23 05:36:40,69 -not-actor-/PID:75834 esrally.rally INFO OS [uname_result(system='Darwin', node='Gunits-MacBook-Air.local', release='23.4.0', version='Darwin Kernel Version 23.4.0: Fri Mar 15 00:19:22 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T8112', machine='arm64')]
2024-05-23 05:36:40,69 -not-actor-/PID:75834 esrally.rally INFO Python [namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=6, releaselevel='final', serial=0), hexversion=50923248, _multiarch='darwin')]
2024-05-23 05:36:40,69 -not-actor-/PID:75834 esrally.rally INFO Rally version [2.10.0]
2024-05-23 05:36:40,69 -not-actor-/PID:75834 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-05-23 05:36:40,70 -not-actor-/PID:75834 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-05-23 05:36:40,70 -not-actor-/PID:75834 esrally.rally INFO Cleaning track dependency directory [/Users/gunitmittal/.rally/libs]...
2024-05-23 05:36:40,86 -not-actor-/PID:75834 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 1183, in dispatch_sub_command
    race(cfg, args.kill_running_processes)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 929, in race
    raise exceptions.RallyError(msg)
esrally.exceptions.RallyError: There are other Rally processes running on this machine (PIDs: [74304, 74305]) but only one Rally benchmark is allowed to run at the same time.

You can use --kill-running-processes flag to kill running processes automatically and allow Rally to continue to run a new benchmark. Otherwise, you need to manually kill them.
2024-05-23 05:36:54,904 -not-actor-/PID:75902 esrally.rally INFO OS [uname_result(system='Darwin', node='Gunits-MacBook-Air.local', release='23.4.0', version='Darwin Kernel Version 23.4.0: Fri Mar 15 00:19:22 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T8112', machine='arm64')]
2024-05-23 05:36:54,904 -not-actor-/PID:75902 esrally.rally INFO Python [namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=6, releaselevel='final', serial=0), hexversion=50923248, _multiarch='darwin')]
2024-05-23 05:36:54,905 -not-actor-/PID:75902 esrally.rally INFO Rally version [2.10.0]
2024-05-23 05:36:54,905 -not-actor-/PID:75902 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-05-23 05:36:54,905 -not-actor-/PID:75902 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-05-23 05:36:54,905 -not-actor-/PID:75902 esrally.rally INFO Cleaning track dependency directory [/Users/gunitmittal/.rally/libs]...
2024-05-23 05:36:54,905 -not-actor-/PID:75902 esrally.rally INFO Killing running Rally processes
2024-05-23 05:36:54,919 -not-actor-/PID:75902 esrally.utils.process INFO Killing lingering process with PID [74304] and command line [['/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python', '/Users/gunitmittal/Desktop/rally/.venv/bin/esrally', 'race', '--distribution-version=7.0.0', '--track=geonames', '--kill-running-processes']].
2024-05-23 05:36:55,921 -not-actor-/PID:75902 esrally.utils.process INFO Killing lingering process with PID [74305] and command line [['/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python', '/Users/gunitmittal/Desktop/rally/.venv/bin/esrally', 'race', '--distribution-version=7.0.0', '--track=geonames', '--kill-running-processes']].
2024-05-23 05:36:56,934 -not-actor-/PID:75902 esrally.rally INFO Actor system already running locally? [False]
2024-05-23 05:36:56,934 -not-actor-/PID:75902 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].
2024-05-23 05:36:56,969 -not-actor-/PID:75912 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)
2024-05-23 05:37:01,968 -not-actor-/PID:75902 esrally.actor ERROR Could not initialize internal actor system.
Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/actor.py", line 263, in bootstrap_actor_system
    return thespian.actors.ActorSystem(system_base, logDefs=log.load_configuration(), capabilities=capabilities)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 635, in __init__
    systemBase = self._startupActorSys(
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 676, in _startupActorSys
    systemBase = sbc(self, logDefs=logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocTCPBase.py", line 28, in __init__
    super(ActorSystemBase, self).__init__(system, logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocCommon.py", line 86, in __init__
    super(multiprocessCommon, self).__init__(system, logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/systemBase.py", line 326, in __init__
    self._startAdmin(self.adminAddr,
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocCommon.py", line 114, in _startAdmin
    raise InvalidActorAddress(adminAddr,
thespian.actors.InvalidActorAddress: ActorAddr-(T|:1900) is not a valid ActorSystem admin
2024-05-23 05:37:01,985 -not-actor-/PID:75902 esrally.rally INFO Falling back to offline actor system.
2024-05-23 05:37:01,996 -not-actor-/PID:75940 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-Q.ThespianQ
2024-05-23 05:37:02,8 -not-actor-/PID:75902 esrally.racecontrol INFO Race id is [269b2b73-0e25-46e1-bb65-669a6621ea47]
2024-05-23 05:37:02,8 -not-actor-/PID:75902 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2024-05-23 05:37:02,8 -not-actor-/PID:75902 esrally.racecontrol INFO Using configured hosts [{'host': 'localhost', 'port': 9200}]
2024-05-23 05:37:05,17 -not-actor-/PID:75902 esrally.rally INFO Attempting to shutdown internal actor system.
2024-05-23 05:37:05,45 -not-actor-/PID:75940 root INFO ---- Actor System shutdown
2024-05-23 05:37:05,48 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:06,54 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:07,59 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:08,61 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:09,66 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:10,73 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:11,76 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:12,78 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:13,82 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:14,88 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:15,94 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:16,97 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:17,100 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:18,105 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:19,107 -not-actor-/PID:75902 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:37:20,111 -not-actor-/PID:75902 esrally.rally WARNING Shutdown timed out. Actor system is still running.
2024-05-23 05:37:20,112 -not-actor-/PID:75902 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 396, in run
    pipeline(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 72, in __call__
    self.target(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 335, in benchmark_only
    return race(cfg, external=True)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 283, in race
    benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True})
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 702, in createActor
    return self._systemBase.newPrimaryActor(actorClass,
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/systemBase.py", line 205, in newPrimaryActor
    raise NoCompatibleSystemForActor(
thespian.actors.NoCompatibleSystemForActor: No compatible ActorSystem could be found for Actor <class 'esrally.racecontrol.BenchmarkActor'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 1183, in dispatch_sub_command
    race(cfg, args.kill_running_processes)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 931, in race
    with_actor_system(racecontrol.run, cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 961, in with_actor_system
    runnable(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 405, in run
    raise exceptions.RallyError("This race ended with a fatal crash.").with_traceback(tb)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 396, in run
    pipeline(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 72, in __call__
    self.target(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 335, in benchmark_only
    return race(cfg, external=True)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 283, in race
    benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True})
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 702, in createActor
    return self._systemBase.newPrimaryActor(actorClass,
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/systemBase.py", line 205, in newPrimaryActor
    raise NoCompatibleSystemForActor(
esrally.exceptions.RallyError: This race ended with a fatal crash.
2024-05-23 05:39:59,26 -not-actor-/PID:76678 esrally.rally INFO OS [uname_result(system='Darwin', node='Gunits-MacBook-Air.local', release='23.4.0', version='Darwin Kernel Version 23.4.0: Fri Mar 15 00:19:22 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T8112', machine='arm64')]
2024-05-23 05:39:59,26 -not-actor-/PID:76678 esrally.rally INFO Python [namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=6, releaselevel='final', serial=0), hexversion=50923248, _multiarch='darwin')]
2024-05-23 05:39:59,26 -not-actor-/PID:76678 esrally.rally INFO Rally version [2.10.0]
2024-05-23 05:39:59,26 -not-actor-/PID:76678 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-05-23 05:39:59,26 -not-actor-/PID:76678 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-05-23 05:39:59,27 -not-actor-/PID:76678 esrally.rally INFO Cleaning track dependency directory [/Users/gunitmittal/.rally/libs]...
2024-05-23 05:39:59,27 -not-actor-/PID:76678 esrally.rally INFO Killing running Rally processes
2024-05-23 05:39:59,43 -not-actor-/PID:76678 esrally.utils.process INFO Killing lingering process with PID [75912] and command line [['/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python', '/Users/gunitmittal/Desktop/rally/.venv/bin/esrally', 'race', '--pipeline=benchmark-only', '--target-hosts=localhost:9200', "--client-options=basic_auth_user:'elastic',basic_auth_password:'changeme'", '--track=geonames', '--challenge=append-fast-with-conflicts', '--kill-running-processes']].
2024-05-23 05:40:00,48 -not-actor-/PID:76678 esrally.utils.process INFO Killing lingering process with PID [75913] and command line [['/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python', '/Users/gunitmittal/Desktop/rally/.venv/bin/esrally', 'race', '--pipeline=benchmark-only', '--target-hosts=localhost:9200', "--client-options=basic_auth_user:'elastic',basic_auth_password:'changeme'", '--track=geonames', '--challenge=append-fast-with-conflicts', '--kill-running-processes']].
2024-05-23 05:40:01,60 -not-actor-/PID:76678 esrally.rally INFO Actor system already running locally? [False]
2024-05-23 05:40:01,60 -not-actor-/PID:76678 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].
2024-05-23 05:40:01,88 -not-actor-/PID:76697 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)
2024-05-23 05:40:01,97 -not-actor-/PID:76678 esrally.racecontrol INFO Race id is [3e7a876a-5791-4794-8275-23a986cd3cfc]
2024-05-23 05:40:01,97 -not-actor-/PID:76678 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2024-05-23 05:40:01,97 -not-actor-/PID:76678 esrally.racecontrol INFO Using configured hosts [{'host': 'localhost', 'port': 9200}]
2024-05-23 05:40:01,99 ActorAddr-(T|:1900)/PID:76697 esrally.actor DEBUG Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 9, 6, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1716442801078'}] match requirements [{'coordinator': True}].
2024-05-23 05:40:01,192 -not-actor-/PID:76678 esrally.racecontrol ERROR A benchmark failure has occurred
2024-05-23 05:40:01,192 -not-actor-/PID:76678 esrally.racecontrol INFO Telling benchmark actor to exit.
2024-05-23 05:40:01,105 ActorAddr-(T|:57863)/PID:76699 esrally.client.factory INFO Creating ES client connected to [{'host': 'localhost', 'port': 9200}] with options [{'timeout': 60, 'basic_auth_user': 'elastic', 'basic_auth_password': '*****'}]
2024-05-23 05:40:01,183 ActorAddr-(T|:57863)/PID:76699 esrally.actor ERROR Error in race control
Traceback (most recent call last):

  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)

  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)

  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 194, in setup
    ) = client.factory.cluster_distribution_version(hosts, client_options)

  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/client/factory.py", line 349, in cluster_distribution_version
    version = es.info()["version"]

  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)

  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/elasticsearch/_sync/client/__init__.py", line 2296, in info
    return self.perform_request(  # type: ignore[return-value]

  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/client/synchronous.py", line 178, in perform_request
    _ProductChecker.raise_error(self._verified_elasticsearch, info_meta, info_body)

  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/client/synchronous.py", line 73, in raise_error
    raise UnsupportedProductError(message, meta=meta, body=body)

elasticsearch.UnsupportedProductError: The client noticed that the server is not Elasticsearch and we do not support this unknown product

2024-05-23 05:40:04,193 -not-actor-/PID:76678 esrally.rally INFO Attempting to shutdown internal actor system.
2024-05-23 05:40:04,197 -not-actor-/PID:76698 root INFO ActorSystem Logging Shutdown
2024-05-23 05:40:04,220 -not-actor-/PID:76697 root INFO ---- Actor System shutdown
2024-05-23 05:40:04,221 -not-actor-/PID:76678 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 05:40:05,223 -not-actor-/PID:76678 esrally.rally INFO Shutdown completed.
2024-05-23 05:40:05,223 -not-actor-/PID:76678 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 1183, in dispatch_sub_command
    race(cfg, args.kill_running_processes)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 931, in race
    with_actor_system(racecontrol.run, cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 961, in with_actor_system
    runnable(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 399, in run
    raise e
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 396, in run
    pipeline(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 72, in __call__
    self.target(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 335, in benchmark_only
    return race(cfg, external=True)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 293, in race
    raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 194, in setup
    ) = client.factory.cluster_distribution_version(hosts, client_options)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/client/factory.py", line 349, in cluster_distribution_version
    version = es.info()["version"]
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/elasticsearch/_sync/client/__init__.py", line 2296, in info
    return self.perform_request(  # type: ignore[return-value]
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/client/synchronous.py", line 178, in perform_request
    _ProductChecker.raise_error(self._verified_elasticsearch, info_meta, info_body)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/client/synchronous.py", line 73, in raise_error
    raise UnsupportedProductError(message, meta=meta, body=body)
elasticsearch.UnsupportedProductError: The client noticed that the server is not Elasticsearch and we do not support this unknown product

2024-05-23 07:41:59,283 -not-actor-/PID:21690 esrally.rally INFO OS [uname_result(system='Darwin', node='Gunits-MacBook-Air.local', release='23.4.0', version='Darwin Kernel Version 23.4.0: Fri Mar 15 00:19:22 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T8112', machine='arm64')]
2024-05-23 07:41:59,284 -not-actor-/PID:21690 esrally.rally INFO Python [namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=6, releaselevel='final', serial=0), hexversion=50923248, _multiarch='darwin')]
2024-05-23 07:41:59,284 -not-actor-/PID:21690 esrally.rally INFO Rally version [2.10.0]
2024-05-23 07:41:59,284 -not-actor-/PID:21690 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-05-23 07:41:59,284 -not-actor-/PID:21690 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-05-23 07:41:59,284 -not-actor-/PID:21690 esrally.rally INFO Cleaning track dependency directory [/Users/gunitmittal/.rally/libs]...
2024-05-23 07:41:59,284 -not-actor-/PID:21690 esrally.rally INFO Killing running Rally processes
2024-05-23 07:41:59,310 -not-actor-/PID:21690 esrally.rally INFO Actor system already running locally? [False]
2024-05-23 07:41:59,310 -not-actor-/PID:21690 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].
2024-05-23 07:41:59,342 -not-actor-/PID:21697 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)
2024-05-23 07:42:04,334 -not-actor-/PID:21690 esrally.actor ERROR Could not initialize internal actor system.
Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/actor.py", line 263, in bootstrap_actor_system
    return thespian.actors.ActorSystem(system_base, logDefs=log.load_configuration(), capabilities=capabilities)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 635, in __init__
    systemBase = self._startupActorSys(
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 676, in _startupActorSys
    systemBase = sbc(self, logDefs=logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocTCPBase.py", line 28, in __init__
    super(ActorSystemBase, self).__init__(system, logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocCommon.py", line 86, in __init__
    super(multiprocessCommon, self).__init__(system, logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/systemBase.py", line 326, in __init__
    self._startAdmin(self.adminAddr,
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocCommon.py", line 114, in _startAdmin
    raise InvalidActorAddress(adminAddr,
thespian.actors.InvalidActorAddress: ActorAddr-(T|:1900) is not a valid ActorSystem admin
2024-05-23 07:42:04,354 -not-actor-/PID:21690 esrally.rally INFO Falling back to offline actor system.
2024-05-23 07:42:04,395 -not-actor-/PID:21720 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-Q.ThespianQ
2024-05-23 07:42:04,405 -not-actor-/PID:21690 esrally.racecontrol INFO Race id is [9d6931d0-d903-435c-a744-361b6e0bfbb3]
2024-05-23 07:42:04,405 -not-actor-/PID:21690 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2024-05-23 07:42:04,405 -not-actor-/PID:21690 esrally.racecontrol INFO Using configured hosts [{'host': 'localhost', 'port': 9200}]
2024-05-23 07:42:07,415 -not-actor-/PID:21690 esrally.rally INFO Attempting to shutdown internal actor system.
2024-05-23 07:42:07,442 -not-actor-/PID:21720 root INFO ---- Actor System shutdown
2024-05-23 07:42:07,444 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:08,451 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:09,453 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:10,456 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:11,462 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:12,464 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:13,466 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:14,468 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:15,470 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:16,475 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:17,481 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:18,488 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:19,492 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:20,498 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:21,504 -not-actor-/PID:21690 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:42:22,510 -not-actor-/PID:21690 esrally.rally WARNING Shutdown timed out. Actor system is still running.
2024-05-23 07:42:22,510 -not-actor-/PID:21690 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 396, in run
    pipeline(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 72, in __call__
    self.target(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 335, in benchmark_only
    return race(cfg, external=True)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 283, in race
    benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True})
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 702, in createActor
    return self._systemBase.newPrimaryActor(actorClass,
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/systemBase.py", line 205, in newPrimaryActor
    raise NoCompatibleSystemForActor(
thespian.actors.NoCompatibleSystemForActor: No compatible ActorSystem could be found for Actor <class 'esrally.racecontrol.BenchmarkActor'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 1183, in dispatch_sub_command
    race(cfg, args.kill_running_processes)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 931, in race
    with_actor_system(racecontrol.run, cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/rally.py", line 961, in with_actor_system
    runnable(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 405, in run
    raise exceptions.RallyError("This race ended with a fatal crash.").with_traceback(tb)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 396, in run
    pipeline(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 72, in __call__
    self.target(cfg)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 335, in benchmark_only
    return race(cfg, external=True)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/racecontrol.py", line 283, in race
    benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True})
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 702, in createActor
    return self._systemBase.newPrimaryActor(actorClass,
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/systemBase.py", line 205, in newPrimaryActor
    raise NoCompatibleSystemForActor(
esrally.exceptions.RallyError: This race ended with a fatal crash.
2024-05-23 07:46:57,273 -not-actor-/PID:22867 esrally.rally INFO OS [uname_result(system='Darwin', node='Gunits-MacBook-Air.local', release='23.4.0', version='Darwin Kernel Version 23.4.0: Fri Mar 15 00:19:22 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T8112', machine='arm64')]
2024-05-23 07:46:57,273 -not-actor-/PID:22867 esrally.rally INFO Python [namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=6, releaselevel='final', serial=0), hexversion=50923248, _multiarch='darwin')]
2024-05-23 07:46:57,273 -not-actor-/PID:22867 esrally.rally INFO Rally version [2.10.0]
2024-05-23 07:46:57,273 -not-actor-/PID:22867 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-05-23 07:46:57,273 -not-actor-/PID:22867 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-05-23 07:46:57,273 -not-actor-/PID:22867 esrally.rally INFO Cleaning track dependency directory [/Users/gunitmittal/.rally/libs]...
2024-05-23 07:46:57,273 -not-actor-/PID:22867 esrally.rally INFO Killing running Rally processes
2024-05-23 07:46:57,299 -not-actor-/PID:22867 esrally.rally INFO Actor system already running locally? [False]
2024-05-23 07:46:57,299 -not-actor-/PID:22867 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].
2024-05-23 07:46:57,315 -not-actor-/PID:22873 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)
2024-05-23 07:47:02,319 -not-actor-/PID:22867 esrally.actor ERROR Could not initialize internal actor system.
Traceback (most recent call last):
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/esrally/actor.py", line 263, in bootstrap_actor_system
    return thespian.actors.ActorSystem(system_base, logDefs=log.load_configuration(), capabilities=capabilities)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 635, in __init__
    systemBase = self._startupActorSys(
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/actors.py", line 676, in _startupActorSys
    systemBase = sbc(self, logDefs=logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocTCPBase.py", line 28, in __init__
    super(ActorSystemBase, self).__init__(system, logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocCommon.py", line 86, in __init__
    super(multiprocessCommon, self).__init__(system, logDefs)
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/systemBase.py", line 326, in __init__
    self._startAdmin(self.adminAddr,
  File "/Users/gunitmittal/Desktop/rally/.venv/lib/python3.9/site-packages/thespian/system/multiprocCommon.py", line 114, in _startAdmin
    raise InvalidActorAddress(adminAddr,
thespian.actors.InvalidActorAddress: ActorAddr-(T|:1900) is not a valid ActorSystem admin
2024-05-23 07:47:02,339 -not-actor-/PID:22867 esrally.rally INFO Falling back to offline actor system.
2024-05-23 07:47:02,354 -not-actor-/PID:22889 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-Q.ThespianQ
2024-05-23 07:47:02,365 -not-actor-/PID:22867 esrally.racecontrol INFO Race id is [a9b727e2-2b1d-4bae-a5eb-8587cf02390e]
2024-05-23 07:47:02,366 -not-actor-/PID:22867 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2024-05-23 07:47:02,369 -not-actor-/PID:22867 esrally.racecontrol INFO Using configured hosts [{'host': 'localhost', 'port': 9200}]
2024-05-23 07:47:05,388 -not-actor-/PID:22867 esrally.rally INFO Attempting to shutdown internal actor system.
2024-05-23 07:47:05,411 -not-actor-/PID:22889 root INFO ---- Actor System shutdown
2024-05-23 07:47:05,413 -not-actor-/PID:22867 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:47:06,414 -not-actor-/PID:22867 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:47:07,419 -not-actor-/PID:22867 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:47:08,425 -not-actor-/PID:22867 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:47:09,429 -not-actor-/PID:22867 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:47:10,433 -not-actor-/PID:22867 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:47:11,436 -not-actor-/PID:22867 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:47:12,438 -not-actor-/PID:22867 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:47:13,444 -not-actor-/PID:22867 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:47:14,447 -not-actor-/PID:22867 esrally.rally INFO Actor system is still running. Waiting...
2024-05-23 07:47:15,454 -not-actor-/PID:22867 esrally.rally INFO Actor system is still running. Waiting...

I am a first time user exploring the elasticsearch tool and was trying to run it locally. Would really appreciate any assistance for this.

system · May 23, 2024, 8:34am

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. See What is OpenSearch and the OpenSearch Dashboard? | Elastic for more details.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

gbanasiak · May 23, 2024, 9:45am

You do not seem to be running Elasticsearch:

elasticsearch.UnsupportedProductError: The client noticed that the server is not Elasticsearch and we do not support this unknown product

Other errors might be due to previous Rally processes left on the host. If --kill-running-processes is not helping, make sure no previous Rally processes are left before running a new Rally instance, and remove them if necessary.