Esrally error - Could not initialize internal actor system

pro.shivansh · July 13, 2020, 2:21pm

Hello everyone, I was running a benchmark-only for a cluster and ran into an error as described below. I would really appreciate it if someone can help me figure out why is this error occurring and a possible resolution.

I am using pyenv to manage my version of python, I used python 3.6.8 and finally 3.8.2 after noticing the requirement on the esrally install page. Got the same error in both cases.

I was running the following command -

 esrally --track-path=<absolute path> --target-hosts=<URL> --pipeline=benchmark-only --client-options="basic_auth_user:'',basic_auth_password:'',verify_certs:false"

I ran the command twice, the second time after it had failed so the errors after 2020-07-10 15:24:05,567 would be related to the time the command was ran the second time.

These are the resulting logs -

2020-07-10 15:23:41,535 -not-actor-/PID:51873 esrally.rally INFO Race id [aaaaa-bbbb-cccc-dddd-eeeeeeee]
2020-07-10 15:23:41,535 -not-actor-/PID:51873 esrally.rally INFO OS [uname_result(system='Darwin', node='xxxxyyyy', release='19.3.0', version='Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64', machine='x86_64', processor='i386')]
2020-07-10 15:23:41,535 -not-actor-/PID:51873 esrally.rally INFO Python [namespace(_multiarch='darwin', cache_tag='cpython-38', hexversion=50856688, name='cpython', version=sys.version_info(major=3, minor=8, micro=2, releaselevel='final', serial=0))]
2020-07-10 15:23:41,535 -not-actor-/PID:51873 esrally.rally INFO Rally version [2.0.0]
2020-07-10 15:23:41,536 -not-actor-/PID:51873 esrally.utils.net INFO Connecting directly to the Internet (no proxy support).
2020-07-10 15:23:42,234 -not-actor-/PID:51873 esrally.rally INFO Detected a working Internet connection.
2020-07-10 15:23:42,301 -not-actor-/PID:51873 esrally.rally INFO Actor system already running locally? [False]
2020-07-10 15:23:42,301 -not-actor-/PID:51873 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': 'xxxx', 'Convention Address.IPv4': 'xxxx'}].
2020-07-10 15:23:42,388 -not-actor-/PID:51873 root WARNING Unable to get address info for address xxxxyyyy (AddressFamily.AF_INET, SocketKind.SOCK_DGRAM, 17, 0): <class 'socket.gaierror'> [Errno 8] nodename nor servname provided, or not known
2020-07-10 15:23:42,390 -not-actor-/PID:51873 root WARNING Unable to get address info for address xxxxyyyy (AddressFamily.AF_INET, SocketKind.SOCK_DGRAM, 17, 0): <class 'socket.gaierror'> [Errno 8] nodename nor servname provided, or not known
2020-07-10 15:23:42,392 -not-actor-/PID:51873 root WARNING Unable to get address info for address xxxxyyyy (AddressFamily.AF_INET, SocketKind.SOCK_DGRAM, 17, AddressInfo.AI_PASSIVE): <class 'socket.gaierror'> [Errno 8] nodename nor servname provided, or not known
2020-07-10 15:23:42,394 -not-actor-/PID:51873 root WARNING Unable to get address info for address xxxxyyyy (AddressFamily.AF_INET, SocketKind.SOCK_DGRAM, 17, AddressInfo.AI_PASSIVE): <class 'socket.gaierror'> [Errno 8] nodename nor servname provided, or not known
2020-07-10 15:23:42,410 -not-actor-/PID:51877 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)
2020-07-10 15:23:47,406 -not-actor-/PID:51873 esrally.actor ERROR Could not initialize internal actor system.
Traceback (most recent call last):
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/actor.py", line 248, in bootstrap_actor_system
    return thespian.actors.ActorSystem(system_base,
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/actors.py", line 635, in __init__
    systemBase = self._startupActorSys(
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/actors.py", line 676, in _startupActorSys
    systemBase = sbc(self, logDefs=logDefs)
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/system/multiprocTCPBase.py", line 28, in __init__
    super(ActorSystemBase, self).__init__(system, logDefs)
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/system/multiprocCommon.py", line 86, in __init__
    super(multiprocessCommon, self).__init__(system, logDefs)
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/system/systemBase.py", line 326, in __init__
    self._startAdmin(self.adminAddr,
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/system/multiprocCommon.py", line 114, in _startAdmin
    raise InvalidActorAddress(adminAddr,
thespian.actors.InvalidActorAddress: ActorAddr-(T|:1900) is not a valid ActorSystem admin
2020-07-10 15:23:47,414 -not-actor-/PID:51873 esrally.rally INFO Falling back to offline actor system.
2020-07-10 15:23:47,415 -not-actor-/PID:51873 esrally.actor INFO Joining already running actor system with system base [multiprocQueueBase].
2020-07-10 15:23:47,438 -not-actor-/PID:51880 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-Q.ThespianQ
2020-07-10 15:23:47,460 -not-actor-/PID:51873 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2020-07-10 15:23:47,460 -not-actor-/PID:51873 esrally.racecontrol INFO Using configured hosts [{'host': 'xxxx', 'port': xxxx, 'use_ssl': True}]
2020-07-10 15:23:47,461 -not-actor-/PID:51873 esrally.actor INFO Joining already running actor system with system base [multiprocQueueBase].
2020-07-10 15:23:50,474 -not-actor-/PID:51873 esrally.rally INFO Attempting to shutdown internal actor system.
2020-07-10 15:23:50,500 -not-actor-/PID:51880 root INFO ---- Actor System shutdown
2020-07-10 15:23:50,502 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:51,506 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:52,512 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:53,517 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:54,521 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:55,524 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:56,529 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:57,533 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:58,537 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:23:59,541 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:24:00,547 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:24:01,552 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:24:02,556 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:24:03,561 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:24:04,564 -not-actor-/PID:51873 esrally.rally INFO Actor system is still running. Waiting...
2020-07-10 15:24:05,567 -not-actor-/PID:51873 esrally.rally WARNING Shutdown timed out. Actor system is still running.
2020-07-10 15:24:05,568 -not-actor-/PID:51873 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 362, in run
    pipeline(cfg)
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 60, in __call__
    self.target(cfg)
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 298, in benchmark_only
    return race(cfg, external=True)
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 241, in race
    benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True})
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/actors.py", line 702, in createActor
    return self._systemBase.newPrimaryActor(actorClass,
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/system/systemBase.py", line 205, in newPrimaryActor
    raise NoCompatibleSystemForActor(
thespian.actors.NoCompatibleSystemForActor: No compatible ActorSystem could be found for Actor <class 'esrally.racecontrol.BenchmarkActor'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/rally.py", line 766, in dispatch_sub_command
    race(cfg)
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/rally.py", line 676, in race
    with_actor_system(racecontrol.run, cfg)
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/rally.py", line 704, in with_actor_system
    runnable(cfg)
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 370, in run
    raise exceptions.RallyError("This race ended with a fatal crash.").with_traceback(tb)
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 362, in run
    pipeline(cfg)
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 60, in __call__
    self.target(cfg)
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 298, in benchmark_only
    return race(cfg, external=True)
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/esrally/racecontrol.py", line 241, in race
    benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True})
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/actors.py", line 702, in createActor
    return self._systemBase.newPrimaryActor(actorClass,
  File "/Users/xxxx/temp/.venv/lib/python3.8/site-packages/thespian/system/systemBase.py", line 205, in newPrimaryActor
    raise NoCompatibleSystemForActor(
esrally.exceptions.RallyError: ('This race ended with a fatal crash.', None)

I am very new to this forum, please excuse me if I missed something or my format is messed up. Suggestions are always welcome.

Evgenia_Badiyanova · July 14, 2020, 3:02pm

Hello Shivansh,

It looks like you issue begins with:

Looks like actor system can't resolve hostname on you local machine. You may need to add it to /etc/hosts.

danielmitterdorfer · July 15, 2020, 6:12am

Hi,

this might as well be because your load driver is connected to the Elasticsearch cluster via a VPN and the actor system cannot derive the appropriate IP address. We have documented a workaround.

However, we strongly advice that the load generator is network-wise very close to the cluster, ideally on a server located in the same rack. Otherwise, you might introduce an accidental bottleneck in your benchmark setup. See our blog post Seven Tips for Better Elasticsearch Benchmarks for more pitfalls to avoid.

Daniel

pro.shivansh · July 24, 2020, 1:35pm

Thank you for this insight Daniel, I was able turn off the VPN and get things moving finally.

system · August 21, 2020, 1:36pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
[Error] Unable to run esrally benchmark in local Elasticsearch rally	2	143	May 23, 2024
Got ActorSystem error when run Esrally Elasticsearch rally	4	1518	March 29, 2017
Esrally failing with [ERROR] Cannot race. Error in race control (NotFoundError(404, 'Not Found', 'Not Found')) Elasticsearch rally	4	729	September 14, 2021
Failure running Rally under OSX Elasticsearch rally	3	1672	July 20, 2017
Your git version is 1.8, but Rally requires at least git 1.9 Elasticsearch rally	4	1406	May 22, 2018

Esrally error - Could not initialize internal actor system

Related topics