Rally does not finish a race (only prepare)

LuisSchaefer · February 25, 2021, 7:29pm

Hi guys,

I am using Rally 1.4.1, Linux RHEL 7.9, offline. If I start a race it prepares the race and that is it.

My Input:

esrally --target-hosts=localhost:9200 --client-options="basic_auth_user:'elastic',basic_auth_password:'password'" --track-path=/root/.rally/benchmarks/tracks/default/geopoint --challenge=append-fast-with-conflicts --offline

Output with Interrupt:

[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[INFO] Preparing for race ...
^C
--------------------------------
[INFO] SUCCESS (took 44 seconds)
--------------------------------

The Log File:

2021-02-25 19:25:05,657 -not-actor-/PID:36388 esrally.rally INFO Race id [9c0d90c5-d308-4fc3-ab10-a9425c6475cb]
2021-02-25 19:25:05,657 -not-actor-/PID:36388 esrally.rally INFO OS [uname_result(system='Linux', node='ile36516', release='3.10.0-1160.11.1.el7.x86_64', version='#1 SMP Mon Nov 30 13:05:31 EST 2020', machine='x86_64', processor='x86_64')]
2021-02-25 19:25:05,658 -not-actor-/PID:36388 esrally.rally INFO Python [namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-36', hexversion=50727152, name='cpython', version=sys.version_info(major=3, minor=6, micro=8, releaselevel='final', serial=0))]
2021-02-25 19:25:05,658 -not-actor-/PID:36388 esrally.rally INFO Rally version [1.4.1]
2021-02-25 19:25:05,658 -not-actor-/PID:36388 esrally.utils.net INFO Connecting directly to the Internet (no proxy support).
2021-02-25 19:25:05,706 -not-actor-/PID:36388 esrally.rally INFO Actor system already running locally? [False]
2021-02-25 19:25:05,706 -not-actor-/PID:36388 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].
2021-02-25 19:25:05,724 -not-actor-/PID:36394 root INFO ++++ Actor System gen (3, 9) started, admin @ ActorAddr-(T|:1900)
2021-02-25 19:25:05,736 -not-actor-/PID:36388 esrally.racecontrol INFO User specified no pipeline. Automatically derived pipeline [from-sources-complete].
2021-02-25 19:25:05,736 -not-actor-/PID:36388 esrally.racecontrol INFO Using configured hosts [{'host': 'localhost', 'port': 9200}]
2021-02-25 19:25:05,737 -not-actor-/PID:36388 esrally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2021-02-25 19:25:05,739 ActorAddr-(T|:1900)/PID:36394 esrally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 6, 8, 'final', 0), 'Thespian Generation': (3, 9), 'Thespian Version': '1614281105716'}] match requirements [{'coordinator': True}].
2021-02-25 19:25:05,739 ActorAddr-(T|:1900)/PID:36394 esrally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 6, 8, 'final', 0), 'Thespian Generation': (3, 9), 'Thespian Version': '1614281105716'}] match requirements [{'coordinator': True}].
2021-02-25 19:25:05,751 ActorAddr-(T|:43379)/PID:36396 esrally.track.loader INFO Reading track specification file [/root/.rally/benchmarks/tracks/default/geopoint/track.json].
2021-02-25 19:25:05,780 ActorAddr-(T|:43379)/PID:36396 esrally.track.loader INFO Final rendered track for '/root/.rally/benchmarks/tracks/default/geopoint/track.json' has been written to '/tmp/tmp9xdy0osj.json'.
2021-02-25 19:25:05,789 ActorAddr-(T|:43379)/PID:36396 esrally.track.loader INFO Loading template [definition for index osmgeopoints in index.json].
2021-02-25 19:25:05,793 ActorAddr-(T|:43379)/PID:36396 esrally.metrics INFO Creating in-memory metrics store
2021-02-25 19:25:05,793 ActorAddr-(T|:43379)/PID:36396 esrally.metrics INFO Opening metrics store for race timestamp=[20210225T192505Z], track=[geopoint], challenge=[append-fast-with-conflicts], car=[['defaults']]
2021-02-25 19:25:05,793 ActorAddr-(T|:43379)/PID:36396 esrally.metrics INFO Creating file race store
2021-02-25 19:25:05,794 ActorAddr-(T|:43379)/PID:36396 esrally.actor INFO Asking mechanic to start the engine.
2021-02-25 19:25:05,794 ActorAddr-(T|:43379)/PID:36396 esrally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 6, 8, 'final', 0), 'Thespian Generation': (3, 9), 'Thespian Version': '1614281105716'}] match requirements [{'coordinator': True}].
2021-02-25 19:25:05,803 ActorAddr-(T|:45053)/PID:36397 esrally.actor INFO Received signal from race control to start engine.
2021-02-25 19:25:05,812 ActorAddr-(T|:45053)/PID:36397 esrally.utils.repo INFO Checking out [master] in [/root/.rally/benchmarks/teams/default] for distribution version [None].
2021-02-25 19:25:05,821 ActorAddr-(T|:45053)/PID:36397 esrally.utils.process INFO Already on 'master'
Your branch is up to date with 'origin/master'.

2021-02-25 19:25:05,822 ActorAddr-(T|:45053)/PID:36397 esrally.utils.repo INFO Rebasing on [master] in [/root/.rally/benchmarks/teams/default] for distribution version [None].
2021-02-25 19:25:05,834 ActorAddr-(T|:45053)/PID:36397 esrally.utils.process INFO Already on 'master'
Your branch is up to date with 'origin/master'.

2021-02-25 19:25:05,841 ActorAddr-(T|:45053)/PID:36397 esrally.utils.process INFO Current branch master is up to date.

2021-02-25 19:25:05,850 ActorAddr-(T|:45053)/PID:36397 esrally.actor INFO Cluster consisting of [{'host': 'localhost', 'port': 9200}] will be provisioned by Rally.
2021-02-25 19:25:05,851 ActorAddr-(T|:45053)/PID:36397 esrally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 6, 8, 'final', 0), 'Thespian Generation': (3, 9), 'Thespian Version': '1614281105716'}] match requirements [{}].
2021-02-25 19:25:46,336 -not-actor-/PID:36388 esrally.racecontrol INFO User has cancelled the benchmark (detected by race control).

Thanks for the help!

Evgenia_Badiyanova · February 26, 2021, 3:13pm

Hello,

Just wanted to double check a few things. When you don't specify --distribution option, the default is to use --revision=current with from-sources pipeline and rally will attempt to build elasticsearch from the source. Could you please see if there is anything in ~/.rally/logs/build.log (same directory as in rally.log.

Thank you,
Evgenia

LuisSchaefer · February 27, 2021, 9:22am

Hi Evgenia,

the file ~/.rally/logs/build.log doesn't exist. How can I fix it?
Some extra information (if needed) Elastic Search + Kibana (Version 7.10.2) are running on docker on the same machine and are up.
Specifing the --distribution-version=7.10.2 didn't help. Probably I am misunderstanding something.

Thanks for your help!

Dale_McDiarmid · March 1, 2021, 1:50pm

Hi Luis
It appears you only gave Rally about 40seconds to install and configure Elasticsearch before interrupting the benchmark? Are you intentionally trying to use an Elasticsearch provisioned by Rally or do you have a local instance running? If the latter, consider --benchmark-only. Finally, if the former does this error persist if you give Rally a few mins?

Evgenia_Badiyanova · March 1, 2021, 2:06pm

Hello,

If you want to benchmark an existing cluster you will want to use --pipeline=benchmark-only.
So your command should look something like:
esrally --target-hosts=localhost:9200 --client-options="basic_auth_user:'elastic',basic_auth_password:'password'" --track-path=/root/.rally/benchmarks/tracks/default/geopoint --challenge=append-fast-with-conflicts --pipeline=benchmark-only --offline

Also, depending on the authentication settings in the cluster you are benchmarking, you may need to set additional flags, like use_ssl and verify_certs:false. Here are some examples.

LuisSchaefer · March 2, 2021, 6:44am

Hi Dale,

the first three attempts ran >999 seconds. The race in my post wasn't optimal to show you this, sorry.

LuisSchaefer · March 2, 2021, 6:47am

Hi Evgenia,

thanks for your help. This solves the problem, but leads to other errors. Hopefully I figure them out with the given examples.
Sorry for (maybe) wasting your time. To make things clear: I am still a student.

Appreciate your help!

system · March 30, 2021, 6:47am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.