# Esrally: error: --target-hosts and --client-options must define the same keys for multi cluster setups

**URL:** <https://discuss.elastic.co/t/esrally-error-target-hosts-and-client-options-must-define-the-same-keys-for-multi-cluster-setups/311780>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [August 10, 2022, 1:30am UTC](https://discuss.elastic.co/t/esrally-error-target-hosts-and-client-options-must-define-the-same-keys-for-multi-cluster-setups/311780 "2022-08-10T01:30:38Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![slowup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slowup/32/99097_2.png) [@slowup](https://discuss.elastic.co/u/slowup)\
**Post date:** [August 10, 2022, 1:30am UTC](https://discuss.elastic.co/t/esrally-error-target-hosts-and-client-options-must-define-the-same-keys-for-multi-cluster-setups/311780/1 "2022-08-10T01:30:38Z")

</div>

**Rally version** elastic/rally latest version

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/4/f4782bfee250df514c7a9334a4e481ffbb6c492e.jpeg)  
The command is as shown above. Because the destination elasticsearch uses xpack, I use client-options to prevent a security exception.  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/2/82082d593e726780b49746ea2782589d4bc334c9.jpeg)

Even after looking at the official documentation, I can't quite understand... The official documentation only says to use it as above. What should I do?

target-hosts is only one  
im use elasticsearch eck!

And target-host is not a multi-cluster

@pquentin

---

<div class="post-metadata">

**Author:** ![Bradley\_Deam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bradley_deam/32/52991_2.png) [@Bradley\_Deam](https://discuss.elastic.co/u/Bradley_Deam)\
**Post date:** [August 10, 2022, 1:38am UTC](https://discuss.elastic.co/t/esrally-error-target-hosts-and-client-options-must-define-the-same-keys-for-multi-cluster-setups/311780/2 "2022-08-10T01:38:31Z")

</div>

Hi @slowup - can you please post the full `--target-hosts` value (redacting any sensitive hostnames/IPs of course).

---

<div class="post-metadata">

**Author:** ![slowup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slowup/32/99097_2.png) [@slowup](https://discuss.elastic.co/u/slowup)\
**Post date:** [August 10, 2022, 1:41am UTC](https://discuss.elastic.co/t/esrally-error-target-hosts-and-client-options-must-define-the-same-keys-for-multi-cluster-setups/311780/3 "2022-08-10T01:41:30Z")

</div>

of course

goods-review-coordinating-service.local-search-engine.svc.cluster.local:9200

I wondered if the dns was a problem, so I tried public, but it didn't work.

@Bradley_Deam

---

<div class="post-metadata">

**Author:** ![Bradley\_Deam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bradley_deam/32/52991_2.png) [@Bradley\_Deam](https://discuss.elastic.co/u/Bradley_Deam)\
**Post date:** [August 10, 2022, 2:53am UTC](https://discuss.elastic.co/t/esrally-error-target-hosts-and-client-options-must-define-the-same-keys-for-multi-cluster-setups/311780/4 "2022-08-10T02:53:42Z")

</div>

> [@slowup](#):
>
> goods-review-coordinating-service.local-search-engine.svc.cluster.local:9200

I can't replicate your error using your invocation locally, either on the latest `esrally` release, or on `master` branch. I even aliased your hostname in `/etc/hosts` to see if there was something funky with the FQDN.

This error usually only appears when you specify something like below where `client-options` doesn't match the corresponding pattern by also including top level dict/json key called `my_cluster_name`;

```auto
esrally race --track geonames --pipeline=benchmark-only --target-hosts='{"my_cluster_name":"goods-review-coordinating-service.local-search-engine.svc.cluster.local:9200"}' --client-options="use_ssl: true, verify_certs: false,basic_auth_user: elastic, basic_auth_password: changeme" --kill-running-processes

```

I assume from the screenshot that you're using K8s, so I wonder if it's possible that the actual command executed is being parsed into something different than what is written in your manifest?

---

<div class="post-metadata">

**Author:** ![slowup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slowup/32/99097_2.png) [@slowup](https://discuss.elastic.co/u/slowup)\
**Post date:** [August 10, 2022, 6:37am UTC](https://discuss.elastic.co/t/esrally-error-target-hosts-and-client-options-must-define-the-same-keys-for-multi-cluster-setups/311780/5 "2022-08-10T06:37:52Z")

</div>

@Bradley_Deam  
command:  
- esrally  
- race  
- --track=geonames  
- --pipeline=benchmark-only  
- --target-hosts=goods-review-coordinating-service.local-search-engine.svc.cluster.local:9200  
- --client-options="use\_ssl:false,verify\_certs:false,basic\_auth\_user:'elastic',basic\_auth\_password:'secret'"

i run command above

Need more information?

---

<div class="post-metadata">

**Author:** ![Bradley\_Deam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bradley_deam/32/52991_2.png) [@Bradley\_Deam](https://discuss.elastic.co/u/Bradley_Deam)\
**Post date:** [August 11, 2022, 12:29am UTC](https://discuss.elastic.co/t/esrally-error-target-hosts-and-client-options-must-define-the-same-keys-for-multi-cluster-setups/311780/6 "2022-08-11T00:29:14Z")

</div>

> [@slowup](#):
>
> - esrally
> - race
> - --track=geonames
> - --pipeline=benchmark-only
> - --target-hosts=goods-review-coordinating-service.local-search-engine.svc.cluster.local:9200
> - --client-options="use\_ssl:false,verify\_certs:false,basic\_auth\_user:'elastic',basic\_auth\_password:'secret'"

Hmm, I still can't reproduce the error you're seeing.

I suspect this is something specific to Docker/K8s. Can you please share your full manifest? I'd like to see if I can replicate this locally with Minikube.

---

<div class="post-metadata">

**Author:** ![slowup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slowup/32/99097_2.png) [@slowup](https://discuss.elastic.co/u/slowup)\
**Post date:** [August 11, 2022, 1:35am UTC](https://discuss.elastic.co/t/esrally-error-target-hosts-and-client-options-must-define-the-same-keys-for-multi-cluster-setups/311780/7 "2022-08-11T01:35:46Z")

</div>

Of course elasticsearch used eck

```auto
apiVersion: batch/v1
kind: Job
metadata:
  name: release-name-elasticsearch-rally
  namespace: local-search-engine
spec:
  template:
    metadata:
      labels:
        app: release-name-elasticsearch-rally
    spec:
      containers:
      - name: esrally
        command:
          - esrally
          - race
          - --track=geonames
          - --pipeline=benchmark-only
          - --target-hosts=goods-review-coordinating-service.local-search-engine.svc.cluster.local:9200
          - --client-options="use_ssl:false,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:'Vx9Lh7CYLN7709Yp2xiw2Q06'"

        image: elastic/rally
        volumeMounts:
        - name: report
          mountPath: /tmp
      restartPolicy: Never
      volumes:
      - name: report
        hostPath:
          path: /tmp

```

---

<div class="post-metadata">

**Author:** ![Bradley\_Deam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bradley_deam/32/52991_2.png) [@Bradley\_Deam](https://discuss.elastic.co/u/Bradley_Deam)\
**Post date:** [August 11, 2022, 4:28am UTC](https://discuss.elastic.co/t/esrally-error-target-hosts-and-client-options-must-define-the-same-keys-for-multi-cluster-setups/311780/8 "2022-08-11T04:28:25Z")

</div>

> [@slowup](#):
>
> `goods-review-coordinating-service.local-search-engine.svc.cluster.local`

Thanks. I was able to reproduce the problem, though I'm still not exactly certain _what_ parsing K8s is doing to the supplied `command` cause the error we see, but I was able to fix it via separating the `command` and `args`:

```auto
apiVersion: batch/v1
kind: Job
metadata:
  name: rally
  namespace: default
spec:
  template:
    metadata:
      labels:
        app: rally
    spec:
      containers:
        - name: esrally
          command: ["esrally"]
          args:
            [
              "race",
              "--track=geonames",
              "--pipeline=benchmark-only",
              "--target-hosts=host.minikube.internal:39200",
              "--client-options='use_ssl:false,verify_certs:false,basic_auth_user:elastic,basic_auth_password:changeme",
            ]
          image: elastic/rally
          volumeMounts:
            - name: report
              mountPath: /rally/.rally/logs
      restartPolicy: Never
      volumes:
        - name: report
          hostPath:
            path: /tmp

```

Give that a shot.

---

<div class="post-metadata">

**Author:** ![slowup](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slowup/32/99097_2.png) [@slowup](https://discuss.elastic.co/u/slowup)\
**Post date:** [August 11, 2022, 4:58am UTC](https://discuss.elastic.co/t/esrally-error-target-hosts-and-client-options-must-define-the-same-keys-for-multi-cluster-setups/311780/9 "2022-08-11T04:58:58Z")

</div>

@Bradley_Deam  
Thanks!! Thanks to you, I think I can finish my work comfortably today!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 8, 2022, 4:59am UTC](https://discuss.elastic.co/t/esrally-error-target-hosts-and-client-options-must-define-the-same-keys-for-multi-cluster-setups/311780/10 "2022-09-08T04:59:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.