# Esrally manometric xpack encrypted es7.2.0

**URL:** <https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [September 3, 2019, 2:19am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748 "2019-09-03T02:19:28Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![rljie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rljie/32/53436_2.png) [@rljie](https://discuss.elastic.co/u/rljie)\
**Post date:** [September 3, 2019, 2:19am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/1 "2019-09-03T02:19:28Z")

</div>

My environment is es7.2.0, esrally1.2.1; Es does xpack encryption and executes the esrally command:  
**esrally --target-hosts=12.19.24.92:9200 --pipeline=benchmark-only --client-options="basic\_auth\_user:'elastic',basic\_auth\_password:'123456',use\_ssl:true,verify\_certs:true,ca\_certs:'/root/config/client-ca.cer',client\_cert:'/root/config/client.cer',client\_key:'/root/config/client.key',timeout:120" --test-mode**

The error was reported as follows:  
**[ERROR] race. ERROR in race control (('Could not connect to your Elasticsearch metrics store. Please check that it is running on host [[https://12.19.24.92](https://12.19.24.92)] at port [9200] or fix the The configuration in [/ root /. Rally/rally. Ini]. ', None))**

I used the curl command to successfully access the es 9200 port ：  
**curl '[https://12.19.24.92:9200/\_cat/nodes?v](https://12.19.24.92:9200/_cat/nodes?v)' --key /home/elasticsearch/client.key --cert /home/elasticsearch/client.cer --cacert /home/elasticsearch/client-ca.cer --user elastic:123456**  
and esrally configuration file is /home/elasticsearch/.rally/rally.ini

---

<div class="post-metadata">

**Author:** ![rljie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rljie/32/53436_2.png) [@rljie](https://discuss.elastic.co/u/rljie)\
**Post date:** [September 3, 2019, 3:54am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/2 "2019-09-03T03:54:09Z")

</div>

Hope someone can discuss this error! Thanks

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [September 3, 2019, 5:15am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/3 "2019-09-03T05:15:40Z")

</div>

Hi,

according to the [docs](https://esrally.readthedocs.io/en/stable/configuration.html#configuration-options) you should add your certificate authority in `~/.rally/rally.ini`. Add the following line in the section `[reporting]`:

```auto
datastore.ssl.certificate_authorities = /home/elasticsearch/client-ca.cer

```

Alternatively you can turn off certificate verification for the metrics store with:

```auto
datastore.ssl.verification_mode = none

```

Also, I noticed you are using the cluster that you are benchmarking also to store metrics which will cause additional load on your cluster. I advise that you instead setup a dedicated cluster to which Rally can send metrics in order to ensure that you will not negatively influence your benchmark.

Daniel

---

<div class="post-metadata">

**Author:** ![rljie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rljie/32/53436_2.png) [@rljie](https://discuss.elastic.co/u/rljie)\
**Post date:** [September 3, 2019, 5:22am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/4 "2019-09-03T05:22:00Z")

</div>

Thank you very much for your reply. I will have a try.

---

<div class="post-metadata">

**Author:** ![rljie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rljie/32/53436_2.png) [@rljie](https://discuss.elastic.co/u/rljie)\
**Post date:** [September 3, 2019, 5:34am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/5 "2019-09-03T05:34:14Z")

</div>

Here's another amazing question !!!  
Execute the esrally command：  
**esrally --target-hosts=12.19.24.92:9200 --pipeline=benchmark-only --client-options="basic\_auth\_user:'elastic',basic\_auth\_password:'123456',use\_ssl:true,verify\_certs:true,ca\_certs:'/home/elasticsearch/client-ca.cer',client\_cert:'/home/elasticsearch/client.cer',client\_key:'/home/elasticsearch/client.key',timeout:120" --test-mode**

The error was reported as follows:  
**[ERROR] Cannot race. Error in race control ([Errno 2] No such file or directory)**

If you put the certificate under /root, use root to execute the esrally command without an error

**Very strange !!!**

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [September 3, 2019, 6:24am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/6 "2019-09-03T06:24:32Z")

</div>

Hi,

this sounds like a file permission problem to me. You can check `~/.rally/logs/rally.log` which provides more detail.

Daniel

---

<div class="post-metadata">

**Author:** ![rljie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rljie/32/53436_2.png) [@rljie](https://discuss.elastic.co/u/rljie)\
**Post date:** [September 3, 2019, 7:20am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/7 "2019-09-03T07:20:53Z")

</div>

Sorry, I just found out that I made a mistake in the initial configuration because I was using root to do it:  
`esrally configure --advanced-config`  
So that leads to a series of problems later!

I have removed /root/.rally directory and re-use elasticsearch user to perform advanced configuration.

Esrally's error report is as follows:  
**[ERROR] Cannot race. Got an unexpected result during benchmarking: [None].**

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [September 3, 2019, 7:35am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/8 "2019-09-03T07:35:05Z")

</div>

Hi,

the command line output is only a first indication that _something_ went wrong. As the next step, please _always_ check the logs. They will contain the full details of what problem has occurred and where.

Daniel

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [September 3, 2019, 8:56am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/10 "2019-09-03T08:56:23Z")

</div>

Hi,

can you please execute the following _before_ you start Rally?

```auto
export THESPLOG_FILE="${HOME}/.rally/logs/actor-system-internal.log"
export THESPLOG_FILE_MAXSIZE=204800
export THESPLOG_THRESHOLD="INFO"

```

And then share the contents of `~/.rally/logs/actor-system-internal.log` (and please share it as preformatted text (use the `</>` in the editor) and not as a screenshot).

Thanks,  
Daniel

---

<div class="post-metadata">

**Author:** ![rljie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rljie/32/53436_2.png) [@rljie](https://discuss.elastic.co/u/rljie)\
**Post date:** [September 3, 2019, 9:07am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/11 "2019-09-03T09:07:39Z")

</div>

Hi，Because I feel that the tips on that part are useful.

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [September 3, 2019, 9:17am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/12 "2019-09-03T09:17:26Z")

</div>

Hi,

> [@rljie](#):
>
> Hi，Because I feel that the tips on that part are useful.

I am sorry but I don't understand what you mean by that. Can you please elaborate?

Thanks,  
Daniel

---

<div class="post-metadata">

**Author:** ![rljie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rljie/32/53436_2.png) [@rljie](https://discuss.elastic.co/u/rljie)\
**Post date:** [September 4, 2019, 1:11am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/14 "2019-09-04T01:11:22Z")

</div>

```
2019-09-04 01:10:23,355 -not-actor-/PID:19588 esrally.rally INFO OS [posix.uname_result(sysname='Linux', nodename='NV-M-GZ-12-19-24-92', release='3.10.0-957.el7.x86_64', version='#1 SMP Thu Nov 8 23:39:32 UTC 2018', machine='x86_64')]
2019-09-04 01:10:23,356 -not-actor-/PID:19588 esrally.rally INFO Python [namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-36', hexversion=50726896, name='cpython', version=sys.version_info(major=3, minor=6, micro=7, releaselevel='final', serial=0))]
2019-09-04 01:10:23,357 -not-actor-/PID:19588 esrally.rally INFO Rally version [1.2.1]
2019-09-04 01:10:23,357 -not-actor-/PID:19588 esrally.utils.net INFO Connecting directly to the Internet (no proxy support).
2019-09-04 01:10:23,956 -not-actor-/PID:19588 esrally.rally INFO Detected a working Internet connection.
2019-09-04 01:10:23,971 -not-actor-/PID:19588 esrally.rally INFO Actor system already running locally? [True]
2019-09-04 01:10:23,971 -not-actor-/PID:19588 esrally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2019-09-04 01:10:25,308 -not-actor-/PID:19588 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2019-09-04 01:10:25,309 -not-actor-/PID:19588 esrally.racecontrol INFO Using configured hosts [{'host': '12.19.24.92', 'port': 9200}]
2019-09-04 01:10:25,309 -not-actor-/PID:19588 esrally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2019-09-04 01:10:25,398 -not-actor-/PID:19588 esrally.racecontrol INFO Telling benchmark actor to exit.
2019-09-04 01:10:25,400 -not-actor-/PID:19588 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 514, in dispatch_sub_command
    race(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 432, in race
    with_actor_system(lambda c: racecontrol.run(c), cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 460, in with_actor_system
    runnable(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 432, in <lambda>
    with_actor_system(lambda c: racecontrol.run(c), cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 427, in run
    raise e
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 424, in run
    pipeline(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 79, in __call__
    self.target(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 352, in benchmark_only
    return race(cfg, external=True)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 306, in race
    raise exceptions.RallyError("Got an unexpected result during benchmarking: [%s]." % str(result))
esrally.exceptions.RallyError: ('Got an unexpected result during benchmarking: [None].', None)
```

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [September 4, 2019, 5:55am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/15 "2019-09-04T05:55:46Z")

</div>

Hi,

this is the output of `~/.rally/logs/rally.log`. Can you please check my recommendations in my earlier post in [Esrally manometric xpack encrypted es7.2.0](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/10) and share `~/.rally/logs/actor-system-internal.log` afterwards?

Thanks,  
Daniel

---

<div class="post-metadata">

**Author:** ![rljie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rljie/32/53436_2.png) [@rljie](https://discuss.elastic.co/u/rljie)\
**Post date:** [September 7, 2019, 4:09am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/16 "2019-09-07T04:09:16Z")

</div>

```
2019-09-07 04:07:32.271697 p19628 I Remote ActorAddr-(T|:44996) closed connection before ack received at ActorAddr-(T|:33583) for TransportIntent(ActorAddr-(T|:44996)-pending-ExpiresIn_0:04:59.990639-<class 'esrally.racecontrol.Setup'>-<esrally.racecontrol.Setup object at 0x7f83fc46ed30>-quit_0:04:59.990579)
2019-09-07 04:07:32.306596 p19628 I ConnRefused to ActorAddr-(T|:44996); declaring as DeadTarget.
2019-09-07 04:07:32.309253 p19628 I ConnRefused to ActorAddr-(T|:44996); declaring as DeadTarget.

```

oh yes，thanks

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [September 9, 2019, 8:26am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/17 "2019-09-09T08:26:24Z")

</div>

Hi,

the only conclusion that I have so far is that it seems to fail during setup but the cause is unclear to me. Can you please confirm that this is the command that you run?

```auto
esrally --target-hosts=12.19.24.92:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'elastic',basic_auth_password:'123456',use_ssl:true,verify_certs:true,ca_certs:'/home/elasticsearch/client-ca.cer',client_cert:'/home/elasticsearch/client.cer',client_key:'/home/elasticsearch/client.key',timeout:120" --test-mode

```

Can you please retry once more without the `--test-mode` argument?

Can you please also share the file `~/.rally/rally.ini` (please remove any credentials before sharing the file)?

Thanks,  
Daniel

---

<div class="post-metadata">

**Author:** ![rljie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rljie/32/53436_2.png) [@rljie](https://discuss.elastic.co/u/rljie)\
**Post date:** [September 10, 2019, 2:32am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/18 "2019-09-10T02:32:44Z")

</div>

It's the same mistake to report without this `--test-mode`.

/home/elasticsearch/.rally/rally.ini ：

```
[meta]
config.version = 17

[system]
env.name = local

[node]
root.dir = /home/elasticsearch/.rally/benchmarks
src.root.dir = /home/elasticsearch/.rally/benchmarks/src

[source]
remote.repo.url = https://github.com/elastic/elasticsearch.git
elasticsearch.src.subdir = elasticsearch

[benchmarks]
local.dataset.cache = /home/elasticsearch/.rally/benchmarks/data

[reporting]
datastore.type = elasticsearch
datastore.host = 12.19.24.92
datastore.port = 9200
datastore.secure = True
datastore.user = elastic
datastore.password = 123456
datastore.ssl.certificate_authorities = /home/elasticsearch/client-ca.cer

[tracks]
default.url = https://github.com/elastic/rally-tracks

[teams]
default.url = https://github.com/elastic/rally-teams

[defaults]
preserve_benchmark_candidate = False

[distributions]
release.cache = true
```

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [September 10, 2019, 5:37am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/19 "2019-09-10T05:37:39Z")

</div>

Hi,

thanks for the config file. All this looks sane to me. Are you running the benchmarks as Unix user `elasticsearch`?

Daniel

---

<div class="post-metadata">

**Author:** ![rljie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rljie/32/53436_2.png) [@rljie](https://discuss.elastic.co/u/rljie)\
**Post date:** [September 10, 2019, 6:16am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/20 "2019-09-10T06:16:35Z")

</div>

Hi, I use the `elasticsearch` user to run commands

---

<div class="post-metadata">

**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\
**Post date:** [September 10, 2019, 8:39am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/21 "2019-09-10T08:39:05Z")

</div>

Hi,

I have a couple of suggestions to help troubleshoot this:

1. Don't benchmark the same Elasticsearch that you are using as a metric store. Could you please use the following `rally.ini` for now?

2. Please ensure there are no left over rally processes; use `ps -ef | grep -i rally` and kill any rally related processes before starting any new benchmarks.

3. Start your esrally command again:

---

<div class="post-metadata">

**Author:** ![rljie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rljie/32/53436_2.png) [@rljie](https://discuss.elastic.co/u/rljie)\
**Post date:** [September 10, 2019, 8:56am UTC](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748/22 "2019-09-10T08:56:53Z")

</div>

Hi,  
Modified `rally.ini` according to your statement. This is esrally command：

```
[elasticsearch@localhost .rally]$ esrally --on-error=abort --target-hosts=12.19.24.92:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'elastic',basic_auth_password:'123456',use_ssl:true,verify_certs:true,ca_certs:'/home/elasticsearch/client-ca.cer',client_cert:'/home/elasticsearch/client.cer',client_key:'/home/elasticsearch/client.key',timeout:120" --test-mode

```

**and** `/home/elasticsearch/.rally/logs/rally.log`

```
2019-09-10 08:47:28,437 -not-actor-/PID:208 esrally.rally INFO OS [posix.uname_result(sysname='Linux', nodename='localhost', release='3.10.0-957.el7.x86_64', version='#1 SMP Thu Nov 8 23:39:32 UTC 2018', machine='x86_64')]
2019-09-10 08:47:28,438 -not-actor-/PID:208 esrally.rally INFO Python [namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-36', hexversion=50726896, name='cpython', version=sys.version_info(major=3, minor=6, micro=7, releaselevel='final', serial=0))]
2019-09-10 08:47:28,439 -not-actor-/PID:208 esrally.rally INFO Rally version [1.2.1]
2019-09-10 08:47:28,439 -not-actor-/PID:208 esrally.utils.net INFO Connecting directly to the Internet (no proxy support).
2019-09-10 08:47:29,277 -not-actor-/PID:208 esrally.rally INFO Detected a working Internet connection.
2019-09-10 08:47:29,285 -not-actor-/PID:208 esrally.rally INFO Actor system already running locally? [True]
2019-09-10 08:47:29,286 -not-actor-/PID:208 esrally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2019-09-10 08:47:29,376 -not-actor-/PID:208 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2019-09-10 08:47:29,377 -not-actor-/PID:208 esrally.racecontrol INFO Using configured hosts [{'host': '12.19.24.92', 'port': 9200}]
2019-09-10 08:47:29,378 -not-actor-/PID:208 esrally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2019-09-10 08:47:29,463 -not-actor-/PID:208 esrally.racecontrol INFO Telling benchmark actor to exit.
2019-09-10 08:47:29,466 -not-actor-/PID:208 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 514, in dispatch_sub_command
    race(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 432, in race
    with_actor_system(lambda c: racecontrol.run(c), cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 460, in with_actor_system
    runnable(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/rally.py", line 432, in <lambda>
    with_actor_system(lambda c: racecontrol.run(c), cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 427, in run
    raise e
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 424, in run
    pipeline(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 79, in __call__
    self.target(cfg)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 352, in benchmark_only
    return race(cfg, external=True)
  File "/usr/local/python3/lib/python3.6/site-packages/esrally/racecontrol.py", line 306, in race
    raise exceptions.RallyError("Got an unexpected result during benchmarking: [%s]." % str(result))
esrally.exceptions.RallyError: ('Got an unexpected result during benchmarking: [None].', None)
```

[Next page](https://discuss.elastic.co/t/esrally-manometric-xpack-encrypted-es7-2-0/197748.md?page=2)