Esrally manometric xpack encrypted es7.2.0

Elastic Stack Elasticsearch
1

My environment is es7.2.0, esrally1.2.1; Es does xpack encryption and executes the esrally command:
esrally --target-hosts=12.19.24.92:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'elastic',basic_auth_password:'123456',use_ssl:true,verify_certs:true,ca_certs:'/root/config/client-ca.cer',client_cert:'/root/config/client.cer',client_key:'/root/config/client.key',timeout:120" --test-mode

The error was reported as follows:
[ERROR] race. ERROR in race control (('Could not connect to your Elasticsearch metrics store. Please check that it is running on host [https://12.19.24.92] at port [9200] or fix the The configuration in [/ root /. Rally/rally. Ini]. ', None))

I used the curl command to successfully access the es 9200 port ：
curl 'https://12.19.24.92:9200/_cat/nodes?v' --key /home/elasticsearch/client.key --cert /home/elasticsearch/client.cer --cacert /home/elasticsearch/client-ca.cer --user elastic:123456
and esrally configuration file is /home/elasticsearch/.rally/rally.ini

2

Hope someone can discuss this error! Thanks

3

Hi,

according to the docs you should add your certificate authority in ~/.rally/rally.ini. Add the following line in the section [reporting]:

datastore.ssl.certificate_authorities = /home/elasticsearch/client-ca.cer

Alternatively you can turn off certificate verification for the metrics store with:

datastore.ssl.verification_mode = none

Also, I noticed you are using the cluster that you are benchmarking also to store metrics which will cause additional load on your cluster. I advise that you instead setup a dedicated cluster to which Rally can send metrics in order to ensure that you will not negatively influence your benchmark.

Daniel

4

Thank you very much for your reply. I will have a try.

5

Here's another amazing question !!!
Execute the esrally command：
esrally --target-hosts=12.19.24.92:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'elastic',basic_auth_password:'123456',use_ssl:true,verify_certs:true,ca_certs:'/home/elasticsearch/client-ca.cer',client_cert:'/home/elasticsearch/client.cer',client_key:'/home/elasticsearch/client.key',timeout:120" --test-mode

The error was reported as follows:
[ERROR] Cannot race. Error in race control ([Errno 2] No such file or directory)

If you put the certificate under /root, use root to execute the esrally command without an error

Very strange !!!

6

Hi,

this sounds like a file permission problem to me. You can check ~/.rally/logs/rally.log which provides more detail.

Daniel

7

Sorry, I just found out that I made a mistake in the initial configuration because I was using root to do it:
esrally configure --advanced-config
So that leads to a series of problems later!

I have removed /root/.rally directory and re-use elasticsearch user to perform advanced configuration.

Esrally's error report is as follows:
[ERROR] Cannot race. Got an unexpected result during benchmarking: [None].

8

Hi,

the command line output is only a first indication that something went wrong. As the next step, please always check the logs. They will contain the full details of what problem has occurred and where.

Daniel

10

Hi,

can you please execute the following before you start Rally?

export THESPLOG_FILE="${HOME}/.rally/logs/actor-system-internal.log"
export THESPLOG_FILE_MAXSIZE=204800
export THESPLOG_THRESHOLD="INFO"

And then share the contents of ~/.rally/logs/actor-system-internal.log (and please share it as preformatted text (use the </> in the editor) and not as a screenshot).

Thanks,
Daniel

11

Hi，Because I feel that the tips on that part are useful.

12

Hi,

I am sorry but I don't understand what you mean by that. Can you please elaborate?

Thanks,
Daniel

14 
2019-09-04 01:10:23,355 -not-actor-/PID:19588 esrally.rally INFO OS [posix.uname_result(sysname='Linux', nodename='NV-M-GZ-12-19-24-92', release='3.10.0-957.el7.x86_64', version='#1 SMP Thu Nov 8 23:39:32 UTC 2018', machine='x86_64')]
2019-09-04 01:10:23,356 -not-actor-/PID:19588 esrally.rally INFO Python [namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-36', hexversion=50726896, name='cpython', version=sys.version_info(major=3, minor=6, micro=7, releaselevel='final', serial=0))]
2019-09-04 01:10:23,357 -not-actor-/PID:19588 esrally.rally INFO Rally version [1.2.1]
2019-09-04 01:10:23,357 -not-actor-/PID:19588 esrally.utils.net INFO Connecting directly to the Internet (no proxy support).
2019-09-04 01:10:23,956 -not-actor-/PID:19588 esrally.rally INFO Detected a working Internet connection.
2019-09-04 01:10:23,971 -not-actor-/PID:19588 esrally.rally INFO Actor system already running locally? [True]
2019-09-04 01:10:23,971 -not-actor-/PID:19588 esrally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2019-09-04 01:10:25,308 -not-actor-/PID:19588 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2019-09-04 01:10:25,309 -not-actor-/PID:19588 esrally.racecontrol INFO Using configured hosts [{'host': '12.19.24.92', 'port': 9200}]
2019-09-04 01:10:25,309 -not-actor-/PID:19588 esrally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2019-09-04 01:10:25,398 -not-actor-/PID:19588 esrally.racecontrol INFO Telling benchmark actor to exit.
2019-09-04 01:10:25,400 -not-actor-/PID:19588 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 514, in dispatch_sub_command
    race(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 432, in race
    with_actor_system(lambda c: racecontrol.run(c), cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 460, in with_actor_system
    runnable(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 432, in <lambda>
    with_actor_system(lambda c: racecontrol.run(c), cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 427, in run
    raise e
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 424, in run
    pipeline(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 79, in __call__
    self.target(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 352, in benchmark_only
    return race(cfg, external=True)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 306, in race
    raise exceptions.RallyError("Got an unexpected result during benchmarking: [%s]." % str(result))
esrally.exceptions.RallyError: ('Got an unexpected result during benchmarking: [None].', None)
15

Hi,

this is the output of ~/.rally/logs/rally.log. Can you please check my recommendations in my earlier post in Esrally manometric xpack encrypted es7.2.0 and share ~/.rally/logs/actor-system-internal.log afterwards?

Thanks,
Daniel

16 
2019-09-07 04:07:32.271697 p19628 I    Remote ActorAddr-(T|:44996) closed connection before ack received at ActorAddr-(T|:33583) for TransportIntent(ActorAddr-(T|:44996)-pending-ExpiresIn_0:04:59.990639-<class 'esrally.racecontrol.Setup'>-<esrally.racecontrol.Setup object at 0x7f83fc46ed30>-quit_0:04:59.990579)
2019-09-07 04:07:32.306596 p19628 I    ConnRefused to ActorAddr-(T|:44996); declaring as DeadTarget.
2019-09-07 04:07:32.309253 p19628 I    ConnRefused to ActorAddr-(T|:44996); declaring as DeadTarget.

oh yes，thanks

17

Hi,

the only conclusion that I have so far is that it seems to fail during setup but the cause is unclear to me. Can you please confirm that this is the command that you run?

esrally --target-hosts=12.19.24.92:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'elastic',basic_auth_password:'123456',use_ssl:true,verify_certs:true,ca_certs:'/home/elasticsearch/client-ca.cer',client_cert:'/home/elasticsearch/client.cer',client_key:'/home/elasticsearch/client.key',timeout:120" --test-mode

Can you please retry once more without the --test-mode argument?

Can you please also share the file ~/.rally/rally.ini (please remove any credentials before sharing the file)?

Thanks,
Daniel

18

It's the same mistake to report without this --test-mode.

/home/elasticsearch/.rally/rally.ini ：

[meta]
config.version = 17

[system]
env.name = local

[node]
root.dir = /home/elasticsearch/.rally/benchmarks
src.root.dir = /home/elasticsearch/.rally/benchmarks/src

[source]
remote.repo.url = https://github.com/elastic/elasticsearch.git
elasticsearch.src.subdir = elasticsearch

[benchmarks]
local.dataset.cache = /home/elasticsearch/.rally/benchmarks/data

[reporting]
datastore.type = elasticsearch
datastore.host = 12.19.24.92
datastore.port = 9200
datastore.secure = True
datastore.user = elastic
datastore.password = 123456
datastore.ssl.certificate_authorities = /home/elasticsearch/client-ca.cer

[tracks]
default.url = https://github.com/elastic/rally-tracks

[teams]
default.url = https://github.com/elastic/rally-teams

[defaults]
preserve_benchmark_candidate = False

[distributions]
release.cache = true
19

Hi,

thanks for the config file. All this looks sane to me. Are you running the benchmarks as Unix user elasticsearch?

Daniel

20

Hi, I use the elasticsearch user to run commands

21

Hi,

I have a couple of suggestions to help troubleshoot this:

  1. Don't benchmark the same Elasticsearch that you are using as a metric store. Could you please use the following rally.ini for now?

    [meta]
config.version = 17

[system]
env.name = local

[node]
root.dir = /home/elasticsearch/.rally/benchmarks
src.root.dir = /home/elasticsearch/.rally/benchmarks/src

[source]
remote.repo.url = https://github.com/elastic/elasticsearch.git
elasticsearch.src.subdir = elasticsearch

[benchmarks]
local.dataset.cache = /home/elasticsearch/.rally/benchmarks/data

[reporting]
datastore.type = in-memory
datastore.host =
datastore.port =
datastore.secure =
datastore.user =
datastore.password =

[tracks]
default.url = https://github.com/elastic/rally-tracks

[teams]
default.url = https://github.com/elastic/rally-teams

[defaults]
preserve_benchmark_candidate = False

[distributions]
release.cache = true

  2. Please ensure there are no left over rally processes; use ps -ef | grep -i rally and kill any rally related processes before starting any new benchmarks.

  3. Start your esrally command again:

    esrally --on-error=abort --target-hosts=12.19.24.92:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'elastic',basic_auth_password:'HIDDEN',use_ssl:true,verify_certs:true,ca_certs:'/home/elasticsearch/client-ca.cer',client_cert:'/home/elasticsearch/client.cer',client_key:'/home/elasticsearch/client.key',timeout:120" --test-mode

    and share the output of the cli command and ~/.rally/logs/rally.log.

22

Hi,
Modified rally.ini according to your statement. This is esrally command：

[elasticsearch@localhost .rally]$ esrally --on-error=abort --target-hosts=12.19.24.92:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'elastic',basic_auth_password:'123456',use_ssl:true,verify_certs:true,ca_certs:'/home/elasticsearch/client-ca.cer',client_cert:'/home/elasticsearch/client.cer',client_key:'/home/elasticsearch/client.key',timeout:120" --test-mode

and /home/elasticsearch/.rally/logs/rally.log

2019-09-10 08:47:28,437 -not-actor-/PID:208 esrally.rally INFO OS [posix.uname_result(sysname='Linux', nodename='localhost', release='3.10.0-957.el7.x86_64', version='#1 SMP Thu Nov 8 23:39:32 UTC 2018', machine='x86_64')]
2019-09-10 08:47:28,438 -not-actor-/PID:208 esrally.rally INFO Python [namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-36', hexversion=50726896, name='cpython', version=sys.version_info(major=3, minor=6, micro=7, releaselevel='final', serial=0))]
2019-09-10 08:47:28,439 -not-actor-/PID:208 esrally.rally INFO Rally version [1.2.1]
2019-09-10 08:47:28,439 -not-actor-/PID:208 esrally.utils.net INFO Connecting directly to the Internet (no proxy support).
2019-09-10 08:47:29,277 -not-actor-/PID:208 esrally.rally INFO Detected a working Internet connection.
2019-09-10 08:47:29,285 -not-actor-/PID:208 esrally.rally INFO Actor system already running locally? [True]
2019-09-10 08:47:29,286 -not-actor-/PID:208 esrally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2019-09-10 08:47:29,376 -not-actor-/PID:208 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2019-09-10 08:47:29,377 -not-actor-/PID:208 esrally.racecontrol INFO Using configured hosts [{'host': '12.19.24.92', 'port': 9200}]
2019-09-10 08:47:29,378 -not-actor-/PID:208 esrally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2019-09-10 08:47:29,463 -not-actor-/PID:208 esrally.racecontrol INFO Telling benchmark actor to exit.
2019-09-10 08:47:29,466 -not-actor-/PID:208 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 514, in dispatch_sub_command
    race(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 432, in race
    with_actor_system(lambda c: racecontrol.run(c), cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 460, in with_actor_system
    runnable(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 432, in <lambda>
    with_actor_system(lambda c: racecontrol.run(c), cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 427, in run
    raise e
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 424, in run
    pipeline(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 79, in __call__
    self.target(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 352, in benchmark_only
    return race(cfg, external=True)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 306, in race
    raise exceptions.RallyError("Got an unexpected result during benchmarking: [%s]." % str(result))
esrally.exceptions.RallyError: ('Got an unexpected result during benchmarking: [None].', None)