[ERROR] Cannot race. Worker [54] has exited prematurely

Elastic Stack Elasticsearch
1

Getting below error while benchmarking with nyc_taxis

Running on kubernetes
1 master - 31 gb heap
3 data - 31 gb heap

System
208 VCPU
mem: 1008 gb

track params

NUMBER_OF_SHARDS=${NUMBER_OF_SHARDS:-10}
NUMBER_OF_REPLICAS=${NUMBER_OF_REPLICAS:-0}
INGEST_PERCENTAGE=${INGEST_PERCENTAGE:-100}
BULK_SIZE=${BULK_SIZE:-10000}
BULK_INDEXING_CLIENTS=${BULK_INDEXING_CLIENTS:-64}
REFRESH_INTERVAL=${REFRESH_INTERVAL:--1}

I am able to execute with 16 clients without any error

2022-06-13 03:10:37,268 -not-actor-/PID:10 esrally.rally INFO Attempting to shutdown internal actor system.
2022-06-13 03:10:37,269 -not-actor-/PID:20 root INFO ActorSystem Logging Shutdown
2022-06-13 03:10:37,290 -not-actor-/PID:19 root INFO ---- Actor System shutdown
2022-06-13 03:10:37,290 -not-actor-/PID:10 esrally.rally INFO Actor system is still running. Waiting...
2022-06-13 03:10:38,291 -not-actor-/PID:10 esrally.rally INFO Shutdown completed.
2022-06-13 03:10:38,291 -not-actor-/PID:10 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/esrally/rally.py", line 975, in dispatch_sub_command
    race(cfg, args.kill_running_processes)
  File "/usr/local/lib/python3.8/site-packages/esrally/rally.py", line 755, in race
    with_actor_system(racecontrol.run, cfg)
  File "/usr/local/lib/python3.8/site-packages/esrally/rally.py", line 785, in with_actor_system
    runnable(cfg)
  File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 373, in run
    raise e
  File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 370, in run
    pipeline(cfg)
  File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 71, in __call__
    self.target(cfg)
  File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 309, in benchmark_only
    return race(cfg, external=True)
  File "/usr/local/lib/python3.8/site-packages/esrally/racecontrol.py", line 267, in race
    raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: Worker [54] has exited prematurely.
2

Hello, do you consistently get this error with 64 indexing clients? Can you please share ~/.rally/logs/rally.log and ~/.rally/logs/actor-system-internal.log?

3

yes i was consistently getting this error. Kindly share where i can send you the logs. in my rally container i only see rally.log

4

Please upload there: Elastic Upload Service : Login. Is it OK if I share the relevant lines in this post?

5

yes, its ok but I am not able to login on above elastic upload service.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.