I want to run a benchmark for my existing 8.9.0 cluster (honestly I've already run tests on that cluster earlier, and everything was great) and i get error right after start.
Command i execute:
esrally race --track=sql --target-hosts=10.64.23.252:9200,10.64.17.157:9200,10.64.25.119:9200 --pipeline=benchmark-only
Error i get:
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
[INFO] Race id is [db13a37e-0bda-4035-af26-1dab884ecaac]
[ERROR] Cannot race. Traceback (most recent call last):
File "/home/tsimmerman/.local/lib/python3.11/site-packages/esrally/actor.py", line 92, in guard
return f(self, msg, sender)
^^^^^^^^^^^^^^^^^^^^
File "/home/tsimmerman/.local/lib/python3.11/site-packages/esrally/driver/driver.py", line 445, in receiveMsg_WakeupMessage
self.logger.exception("Worker failed. Notifying parent...", exc_info=e)
File "/usr/local/lib/python3.11/logging/__init__.py", line 1524, in exception
self.error(msg, *args, exc_info=exc_info, **kwargs)
File "/usr/local/lib/python3.11/logging/__init__.py", line 1518, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/local/lib/python3.11/logging/__init__.py", line 1634, in _log
self.handle(record)
File "/usr/local/lib/python3.11/logging/__init__.py", line 1644, in handle
self.callHandlers(record)
File "/usr/local/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
hdlr.handle(record)
File "/home/tsimmerman/.local/lib/python3.11/site-packages/thespian/system/logdirector.py", line 129, in handle
excinfo = traceback.format_exception(record.exc_info[0],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/traceback.py", line 139, in format_exception
te = TracebackException(type(value), value, tb, limit=limit, compact=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/traceback.py", line 724, in __init__
cause = TracebackException(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/traceback.py", line 698, in __init__
self.__notes__ = getattr(exc_value, '__notes__', None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/tempfile.py", line 477, in __getattr__
file = self.__dict__['file']
~~~~~~~~~~~~~^^^^^^^^
KeyError: 'file'
Getting further help:
*********************
* Check the log files in /home/tsimmerman/.rally/logs for errors.
* Read the documentation at https://esrally.readthedocs.io/en/2.9.0/.
* Ask a question on the forum at https://discuss.elastic.co/tags/c/elastic-stack/elasticsearch/rally.
* Raise an issue at https://github.com/elastic/rally/issues and include the log files in /home/tsimmerman/.rally/logs.
--------------------------------
[INFO] FAILURE (took 27 seconds)
--------------------------------
Log file fillment:
2023-09-04 11:31:10,158 ActorAddr-(T|:43157)/PID:32763 esrally.actor ERROR Main driver received a fatal exception from a load generator. Shutting down.
2023-09-04 11:31:10,160 -not-actor-/PID:32722 esrally.racecontrol ERROR A benchmark failure has occurred
2023-09-04 11:31:10,160 -not-actor-/PID:32722 esrally.racecontrol INFO Telling benchmark actor to exit.
2023-09-04 11:31:10,159 ActorAddr-(T|:42791)/PID:32742 esrally.actor INFO Received a benchmark failure from [ActorAddr-(T|:43157)] and will forward it now.
2023-09-04 11:31:10,159 ActorAddr-(T|:43157)/PID:32763 esrally.metrics INFO Closing metrics store.
2023-09-04 11:31:10,173 ActorAddr-(T|:43157)/PID:32763 esrally.actor INFO Main driver received ActorExitRequest and will terminate all load generators.
2023-09-04 11:31:13,167 -not-actor-/PID:32722 esrally.rally INFO Attempting to shutdown internal actor system.
2023-09-04 11:31:13,168 -not-actor-/PID:32741 root INFO ActorSystem Logging Shutdown
2023-09-04 11:31:13,171 -not-actor-/PID:32722 esrally.rally INFO Actor system is still running. Waiting...
2023-09-04 11:31:13,171 -not-actor-/PID:32740 root INFO ---- Actor System shutdown
2023-09-04 11:31:14,171 -not-actor-/PID:32722 esrally.rally INFO Shutdown completed.
2023-09-04 11:31:14,171 -not-actor-/PID:32722 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
File "/home/tsimmerman/.local/lib/python3.11/site-packages/esrally/rally.py", line 1172, in dispatch_sub_command
race(cfg, args.kill_running_processes)
File "/home/tsimmerman/.local/lib/python3.11/site-packages/esrally/rally.py", line 920, in race
with_actor_system(racecontrol.run, cfg)
File "/home/tsimmerman/.local/lib/python3.11/site-packages/esrally/rally.py", line 950, in with_actor_system
runnable(cfg)
File "/home/tsimmerman/.local/lib/python3.11/site-packages/esrally/racecontrol.py", line 396, in run
raise e
File "/home/tsimmerman/.local/lib/python3.11/site-packages/esrally/racecontrol.py", line 393, in run
pipeline(cfg)
File "/home/tsimmerman/.local/lib/python3.11/site-packages/esrally/racecontrol.py", line 72, in __call__
self.target(cfg)
File "/home/tsimmerman/.local/lib/python3.11/site-packages/esrally/racecontrol.py", line 332, in benchmark_only
return race(cfg, external=True)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tsimmerman/.local/lib/python3.11/site-packages/esrally/racecontrol.py", line 290, in race
raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: Traceback (most recent call last):
File "/home/tsimmerman/.local/lib/python3.11/site-packages/esrally/actor.py", line 92, in guard
return f(self, msg, sender)
^^^^^^^^^^^^^^^^^^^^
File "/home/tsimmerman/.local/lib/python3.11/site-packages/esrally/driver/driver.py", line 445, in receiveMsg_WakeupMessage
self.logger.exception("Worker failed. Notifying parent...", exc_info=e)
File "/usr/local/lib/python3.11/logging/__init__.py", line 1524, in exception
self.error(msg, *args, exc_info=exc_info, **kwargs)
File "/usr/local/lib/python3.11/logging/__init__.py", line 1518, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/local/lib/python3.11/logging/__init__.py", line 1634, in _log
self.handle(record)
File "/usr/local/lib/python3.11/logging/__init__.py", line 1644, in handle
self.callHandlers(record)
File "/usr/local/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
hdlr.handle(record)
File "/home/tsimmerman/.local/lib/python3.11/site-packages/thespian/system/logdirector.py", line 129, in handle
excinfo = traceback.format_exception(record.exc_info[0],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/traceback.py", line 139, in format_exception
te = TracebackException(type(value), value, tb, limit=limit, compact=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/traceback.py", line 724, in __init__
cause = TracebackException(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/traceback.py", line 698, in __init__
self.__notes__ = getattr(exc_value, '__notes__', None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/tempfile.py", line 477, in __getattr__
file = self.__dict__['file']
~~~~~~~~~~~~~^^^^^^^^
KeyError: 'file'
I already tried to reinstall rally, and that didn't help.
One week before it worked fine, don't know what was happened