Hope someone can discuss this error! Thanks
Hi,
according to the docs you should add your certificate authority in ~/.rally/rally.ini
. Add the following line in the section [reporting]
:
datastore.ssl.certificate_authorities = /home/elasticsearch/client-ca.cer
Alternatively you can turn off certificate verification for the metrics store with:
datastore.ssl.verification_mode = none
Also, I noticed you are using the cluster that you are benchmarking also to store metrics which will cause additional load on your cluster. I advise that you instead setup a dedicated cluster to which Rally can send metrics in order to ensure that you will not negatively influence your benchmark.
Daniel
Thank you very much for your reply. I will have a try.
Here's another amazing question !!!
Execute the esrally command:
esrally --target-hosts=12.19.24.92:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'elastic',basic_auth_password:'123456',use_ssl:true,verify_certs:true,ca_certs:'/home/elasticsearch/client-ca.cer',client_cert:'/home/elasticsearch/client.cer',client_key:'/home/elasticsearch/client.key',timeout:120" --test-mode
The error was reported as follows:
[ERROR] Cannot race. Error in race control ([Errno 2] No such file or directory)
If you put the certificate under /root, use root to execute the esrally command without an error
Very strange !!!
Hi,
this sounds like a file permission problem to me. You can check ~/.rally/logs/rally.log
which provides more detail.
Daniel
Sorry, I just found out that I made a mistake in the initial configuration because I was using root to do it:
esrally configure --advanced-config
So that leads to a series of problems later!
I have removed /root/.rally directory and re-use elasticsearch user to perform advanced configuration.
Esrally's error report is as follows:
[ERROR] Cannot race. Got an unexpected result during benchmarking: [None].
Hi,
the command line output is only a first indication that something went wrong. As the next step, please always check the logs. They will contain the full details of what problem has occurred and where.
Daniel
Hi,
can you please execute the following before you start Rally?
export THESPLOG_FILE="${HOME}/.rally/logs/actor-system-internal.log"
export THESPLOG_FILE_MAXSIZE=204800
export THESPLOG_THRESHOLD="INFO"
And then share the contents of ~/.rally/logs/actor-system-internal.log
(and please share it as preformatted text (use the </>
in the editor) and not as a screenshot).
Thanks,
Daniel
Hi,Because I feel that the tips on that part are useful.
Hi,
I am sorry but I don't understand what you mean by that. Can you please elaborate?
Thanks,
Daniel
2019-09-04 01:10:23,355 -not-actor-/PID:19588 esrally.rally INFO OS [posix.uname_result(sysname='Linux', nodename='NV-M-GZ-12-19-24-92', release='3.10.0-957.el7.x86_64', version='#1 SMP Thu Nov 8 23:39:32 UTC 2018', machine='x86_64')]
2019-09-04 01:10:23,356 -not-actor-/PID:19588 esrally.rally INFO Python [namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-36', hexversion=50726896, name='cpython', version=sys.version_info(major=3, minor=6, micro=7, releaselevel='final', serial=0))]
2019-09-04 01:10:23,357 -not-actor-/PID:19588 esrally.rally INFO Rally version [1.2.1]
2019-09-04 01:10:23,357 -not-actor-/PID:19588 esrally.utils.net INFO Connecting directly to the Internet (no proxy support).
2019-09-04 01:10:23,956 -not-actor-/PID:19588 esrally.rally INFO Detected a working Internet connection.
2019-09-04 01:10:23,971 -not-actor-/PID:19588 esrally.rally INFO Actor system already running locally? [True]
2019-09-04 01:10:23,971 -not-actor-/PID:19588 esrally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2019-09-04 01:10:25,308 -not-actor-/PID:19588 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2019-09-04 01:10:25,309 -not-actor-/PID:19588 esrally.racecontrol INFO Using configured hosts [{'host': '12.19.24.92', 'port': 9200}]
2019-09-04 01:10:25,309 -not-actor-/PID:19588 esrally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2019-09-04 01:10:25,398 -not-actor-/PID:19588 esrally.racecontrol INFO Telling benchmark actor to exit.
2019-09-04 01:10:25,400 -not-actor-/PID:19588 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 514, in dispatch_sub_command
race(cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 432, in race
with_actor_system(lambda c: racecontrol.run(c), cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 460, in with_actor_system
runnable(cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 432, in <lambda>
with_actor_system(lambda c: racecontrol.run(c), cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 427, in run
raise e
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 424, in run
pipeline(cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 79, in __call__
self.target(cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 352, in benchmark_only
return race(cfg, external=True)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 306, in race
raise exceptions.RallyError("Got an unexpected result during benchmarking: [%s]." % str(result))
esrally.exceptions.RallyError: ('Got an unexpected result during benchmarking: [None].', None)
Hi,
this is the output of ~/.rally/logs/rally.log
. Can you please check my recommendations in my earlier post in Esrally manometric xpack encrypted es7.2.0 and share ~/.rally/logs/actor-system-internal.log
afterwards?
Thanks,
Daniel
2019-09-07 04:07:32.271697 p19628 I Remote ActorAddr-(T|:44996) closed connection before ack received at ActorAddr-(T|:33583) for TransportIntent(ActorAddr-(T|:44996)-pending-ExpiresIn_0:04:59.990639-<class 'esrally.racecontrol.Setup'>-<esrally.racecontrol.Setup object at 0x7f83fc46ed30>-quit_0:04:59.990579)
2019-09-07 04:07:32.306596 p19628 I ConnRefused to ActorAddr-(T|:44996); declaring as DeadTarget.
2019-09-07 04:07:32.309253 p19628 I ConnRefused to ActorAddr-(T|:44996); declaring as DeadTarget.
oh yes,thanks
Hi,
the only conclusion that I have so far is that it seems to fail during setup but the cause is unclear to me. Can you please confirm that this is the command that you run?
esrally --target-hosts=12.19.24.92:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'elastic',basic_auth_password:'123456',use_ssl:true,verify_certs:true,ca_certs:'/home/elasticsearch/client-ca.cer',client_cert:'/home/elasticsearch/client.cer',client_key:'/home/elasticsearch/client.key',timeout:120" --test-mode
Can you please retry once more without the --test-mode
argument?
Can you please also share the file ~/.rally/rally.ini
(please remove any credentials before sharing the file)?
Thanks,
Daniel
It's the same mistake to report without this --test-mode
.
/home/elasticsearch/.rally/rally.ini :
[meta]
config.version = 17
[system]
env.name = local
[node]
root.dir = /home/elasticsearch/.rally/benchmarks
src.root.dir = /home/elasticsearch/.rally/benchmarks/src
[source]
remote.repo.url = https://github.com/elastic/elasticsearch.git
elasticsearch.src.subdir = elasticsearch
[benchmarks]
local.dataset.cache = /home/elasticsearch/.rally/benchmarks/data
[reporting]
datastore.type = elasticsearch
datastore.host = 12.19.24.92
datastore.port = 9200
datastore.secure = True
datastore.user = elastic
datastore.password = 123456
datastore.ssl.certificate_authorities = /home/elasticsearch/client-ca.cer
[tracks]
default.url = https://github.com/elastic/rally-tracks
[teams]
default.url = https://github.com/elastic/rally-teams
[defaults]
preserve_benchmark_candidate = False
[distributions]
release.cache = true
Hi,
thanks for the config file. All this looks sane to me. Are you running the benchmarks as Unix user elasticsearch
?
Daniel
Hi, I use the elasticsearch
user to run commands
Hi,
I have a couple of suggestions to help troubleshoot this:
-
Don't benchmark the same Elasticsearch that you are using as a metric store. Could you please use the following
rally.ini
for now?[meta] config.version = 17 [system] env.name = local [node] root.dir = /home/elasticsearch/.rally/benchmarks src.root.dir = /home/elasticsearch/.rally/benchmarks/src [source] remote.repo.url = https://github.com/elastic/elasticsearch.git elasticsearch.src.subdir = elasticsearch [benchmarks] local.dataset.cache = /home/elasticsearch/.rally/benchmarks/data [reporting] datastore.type = in-memory datastore.host = datastore.port = datastore.secure = datastore.user = datastore.password = [tracks] default.url = https://github.com/elastic/rally-tracks [teams] default.url = https://github.com/elastic/rally-teams [defaults] preserve_benchmark_candidate = False [distributions] release.cache = true
-
Please ensure there are no left over rally processes; use
ps -ef | grep -i rally
and kill any rally related processes before starting any new benchmarks. -
Start your esrally command again:
esrally --on-error=abort --target-hosts=12.19.24.92:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'elastic',basic_auth_password:'HIDDEN',use_ssl:true,verify_certs:true,ca_certs:'/home/elasticsearch/client-ca.cer',client_cert:'/home/elasticsearch/client.cer',client_key:'/home/elasticsearch/client.key',timeout:120" --test-mode
and share the output of the cli command and
~/.rally/logs/rally.log
.
Hi,
Modified rally.ini
according to your statement. This is esrally command:
[elasticsearch@localhost .rally]$ esrally --on-error=abort --target-hosts=12.19.24.92:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'elastic',basic_auth_password:'123456',use_ssl:true,verify_certs:true,ca_certs:'/home/elasticsearch/client-ca.cer',client_cert:'/home/elasticsearch/client.cer',client_key:'/home/elasticsearch/client.key',timeout:120" --test-mode
and /home/elasticsearch/.rally/logs/rally.log
2019-09-10 08:47:28,437 -not-actor-/PID:208 esrally.rally INFO OS [posix.uname_result(sysname='Linux', nodename='localhost', release='3.10.0-957.el7.x86_64', version='#1 SMP Thu Nov 8 23:39:32 UTC 2018', machine='x86_64')]
2019-09-10 08:47:28,438 -not-actor-/PID:208 esrally.rally INFO Python [namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-36', hexversion=50726896, name='cpython', version=sys.version_info(major=3, minor=6, micro=7, releaselevel='final', serial=0))]
2019-09-10 08:47:28,439 -not-actor-/PID:208 esrally.rally INFO Rally version [1.2.1]
2019-09-10 08:47:28,439 -not-actor-/PID:208 esrally.utils.net INFO Connecting directly to the Internet (no proxy support).
2019-09-10 08:47:29,277 -not-actor-/PID:208 esrally.rally INFO Detected a working Internet connection.
2019-09-10 08:47:29,285 -not-actor-/PID:208 esrally.rally INFO Actor system already running locally? [True]
2019-09-10 08:47:29,286 -not-actor-/PID:208 esrally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2019-09-10 08:47:29,376 -not-actor-/PID:208 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2019-09-10 08:47:29,377 -not-actor-/PID:208 esrally.racecontrol INFO Using configured hosts [{'host': '12.19.24.92', 'port': 9200}]
2019-09-10 08:47:29,378 -not-actor-/PID:208 esrally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2019-09-10 08:47:29,463 -not-actor-/PID:208 esrally.racecontrol INFO Telling benchmark actor to exit.
2019-09-10 08:47:29,466 -not-actor-/PID:208 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 514, in dispatch_sub_command
race(cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 432, in race
with_actor_system(lambda c: racecontrol.run(c), cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 460, in with_actor_system
runnable(cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 432, in <lambda>
with_actor_system(lambda c: racecontrol.run(c), cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 427, in run
raise e
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 424, in run
pipeline(cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 79, in __call__
self.target(cfg)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 352, in benchmark_only
return race(cfg, external=True)
File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 306, in race
raise exceptions.RallyError("Got an unexpected result during benchmarking: [%s]." % str(result))
esrally.exceptions.RallyError: ('Got an unexpected result during benchmarking: [None].', None)
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.