[ERROR] Cannot race. This race ended with a fatal crash

Hi,

Am just getting started with rally and got this error when doing
esrally race --distribution-version=6.8.0 --track=geopoint --challenge=append-fast-with-conflicts

Steps followed as per https://esrally.readthedocs.io/en/stable/race.html-

  1. esrally list tracks
  2. esrally race --distribution-version=6.8.0 --track=geopoint --challenge=append-fast-with-conflicts

Error :
[INFO] Race id is [22704670-dfba-4248-8b9c-9c2177e6bd9e]
INFO:esrally.actor:Checking capabilities [{'Thespian ActorSystem Name': 'multiprocQueueBase', 'Thespian ActorSystem Version': 1, 'Python Version': (3, 9, 6, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1668186421200'}] against requirements [{'coordinator': True}] failed.
[WARNING] Could not terminate all internal processes within timeout. Please check and force-terminate all Rally processes.
[ERROR] Cannot race. This race ended with a fatal crash.

Getting further help:


Logs :

2022-11-11 16:44:15,933 -not-actor-/PID:37013 esrally.rally INFO Falling back to offline actor system.
2022-11-11 16:44:15,933 -not-actor-/PID:37013 esrally.actor INFO Joining already running actor system with system base [multiprocQueueBase].
2022-11-11 16:44:16,4 -not-actor-/PID:37042 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-Q.ThespianQ
2022-11-11 16:44:16,25 -not-actor-/PID:37013 esrally.racecontrol INFO Race id is [569093d2-9506-4a5d-bf5e-3d3b9a678503]
2022-11-11 16:44:16,25 -not-actor-/PID:37013 esrally.racecontrol INFO User specified no pipeline. Automatically derived pipeline [from-distribution].
2022-11-11 16:44:16,26 -not-actor-/PID:37013 esrally.racecontrol INFO Setting default host to [127.0.0.1:39200]
2022-11-11 16:44:16,27 -not-actor-/PID:37013 esrally.actor INFO Joining already running actor system with system base [multiprocQueueBase].
2022-11-11 16:44:19,35 -not-actor-/PID:37013 esrally.rally INFO Attempting to shutdown internal actor system.
2022-11-11 16:44:19,59 -not-actor-/PID:37042 root INFO ---- Actor System shutdown
2022-11-11 16:44:19,61 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:20,63 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:21,68 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:22,72 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:23,76 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:24,81 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:25,84 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:26,88 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:27,93 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:28,99 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:29,100 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:30,113 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:31,116 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:32,120 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:33,124 -not-actor-/PID:37013 esrally.rally INFO Actor system is still running. Waiting...
2022-11-11 16:44:34,129 -not-actor-/PID:37013 esrally.rally WARNING Shutdown timed out. Actor system is still running.
2022-11-11 16:44:34,129 -not-actor-/PID:37013 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/Library/Python/3.9/site-packages/esrally/racecontrol.py", line 370, in run
    pipeline(cfg)
  File "/Library/Python/3.9/site-packages/esrally/racecontrol.py", line 71, in __call__
    self.target(cfg)
  File "/Library/Python/3.9/site-packages/esrally/racecontrol.py", line 302, in from_distribution
    return race(cfg, distribution=True)
  File "/Library/Python/3.9/site-packages/esrally/racecontrol.py", line 257, in race
    benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True})
  File "/Users/psahitya/Library/Python/3.9/lib/python/site-packages/thespian/actors.py", line 702, in createActor
    return self._systemBase.newPrimaryActor(actorClass,
  File "/Users/psahitya/Library/Python/3.9/lib/python/site-packages/thespian/system/systemBase.py", line 205, in newPrimaryActor
    raise NoCompatibleSystemForActor(
thespian.actors.NoCompatibleSystemForActor: No compatible ActorSystem could be found for Actor <class 'esrally.racecontrol.BenchmarkActor'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Python/3.9/site-packages/esrally/rally.py", line 1107, in dispatch_sub_command
    race(cfg, args.kill_running_processes)
  File "/Library/Python/3.9/site-packages/esrally/rally.py", line 865, in race
    with_actor_system(racecontrol.run, cfg)
  File "/Library/Python/3.9/site-packages/esrally/rally.py", line 895, in with_actor_system
    runnable(cfg)
  File "/Library/Python/3.9/site-packages/esrally/racecontrol.py", line 379, in run
    raise exceptions.RallyError("This race ended with a fatal crash.").with_traceback(tb)
  File "/Library/Python/3.9/site-packages/esrally/racecontrol.py", line 370, in run
    pipeline(cfg)
  File "/Library/Python/3.9/site-packages/esrally/racecontrol.py", line 71, in __call__
    self.target(cfg)
  File "/Library/Python/3.9/site-packages/esrally/racecontrol.py", line 302, in from_distribution
    return race(cfg, distribution=True)

Any help appreciated

Hi Sahitya, welcome to the Elastic community. It looks like there could possibly be processes leftover from another Rally run on your system. Can you try executing Rally with --kill-running-processes? The --test-mode flag can be used for fast iteration and troubleshooting:

esrally race --distribution-version=6.8.0 --track=geopoint --challenge=append-fast-with-conflicts --test-mode --kill-running-processes

If you have further questions, could you be sure to include the full rally.log?

Thank you,
Jason

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.