Esrally error in k8s cluster and not able to deploy

Hi All,

I am trying to deploy esrally in kubernetes cluster and below is the manifest I have created,

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: esrally-claim
spec:
  storageClassName: db
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
kind: ConfigMap 
apiVersion: v1 
metadata:
  name: esrally-trackparams-cfg
data:
  params.json: |
    {
       "number_of_shards": 20,
       "number_of_replicas": 0,
       "ingest_percentage": 100,
       "bulk_size": 10000,
       "bulk_indexing_clients": 8,
       "index_settings": { "refresh_interval":-1 }
    }  
---
apiVersion: batch/v1
kind: Job
metadata:
  name: esrally
spec:
  template:
    spec:
      initContainers:
      - name: logrotate
        image: ubuntu:18.04
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh"]
        args: ["-c", "mv /data/logs/rally.log /data/logs/rally.log.$(date +%s) || true"]
        volumeMounts:
        - name: esrally-data
          mountPath: /data/
      containers:
      - name: esrally
        image: elastic/rally:2.2.0
        imagePullPolicy: IfNotPresent
        args: ['--track=nyc_taxis',
               '--pipeline=benchmark-only',
               '--challenge=append-no-conflicts-index-only',
               '--client-options="use_ssl:false,verify_certs:false"',
               '--track-params=/rally/params.json',
               '--target-hosts=https://admin:XXXX@es-client-service:9200']
        volumeMounts:
        - name: esrally-data
          mountPath: /rally/.rally
        - name: trackparams-cfg-vol
          mountPath: /rally/params.json
          subPath: params.json
        imagePullPolicy: Always
      restartPolicy: Never
      volumes:
      - name: esrally-data
        persistentVolumeClaim:
          claimName: esrally-claim
      - name: trackparams-cfg-vol
        configMap:
          name: esrally-trackparams-cfg

When I am trying to bring up the pod, I am getting the below error in logs,

[root@k8master1 prometheus-grafana]# kubectl logs esrally-4gv2z -n es -f
usage: esrally [-h] [--version]
{race,list,info,create-track,generate,compare,download,install,start,stop}
...
esrally: error: unrecognized arguments: --track=nyc_taxis --pipeline=benchmark-only --challenge=append-no-conflicts-index-only --client-options="use_ssl:false,verify_certs:false" --track-params=/rally/params.json --target-hosts=https://admin:XXXX@es-client-service:9200

Please correct me if I am doing anything wrong and share your thoughts it would be really helpful.

Thanks,
Ganeshbabu R

Hey @r.ganeshbabu !

I think you're just missing the race subcommand in your container's arguments, e.g.:

  args: [      'race',
               '--track=nyc_taxis',
               '--pipeline=benchmark-only',
               '--challenge=append-no-conflicts-index-only',
               '--client-options="use_ssl:false,verify_certs:false"',
               '--track-params=/rally/params.json',
               '--target-hosts=https://admin:XXXX@es-client-service:9200']

Can you give that a go and let me know if it works?

Yes @Bradley_Deam

You are right I missed the race subcommand,

I tried redeploying it and I got this in logs,

[root@k8master1 spark_helm]# kubectl logs esrally-7jmlm -n es -f


    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled.
[ERROR] Cannot race. not enough values to unpack (expected 2, got 1).

Getting further help:
*********************
* Check the log files in /rally/.rally/logs for errors.
* Read the documentation at https://esrally.readthedocs.io/en/2.2.0/.
* Ask a question on the forum at https://discuss.elastic.co/tags/c/elastic-stack/elasticsearch/rally.
* Raise an issue at https://github.com/elastic/rally/issues and include the log files in /rally/.rally/logs

Is this error due to esrally not able to find out the tracks?

also I have tried offline mode in my args,

args: ['race',
               '--track=nyc_taxis',
               '--test-mode',
               '--offline',
               '--pipeline=benchmark-only',
               '--challenge=append-no-conflicts-index-only',
               '--client-options="use_ssl:true,verify_certs:true,ca_certs=/data/certificates/cacert.pem"',
               '--track-params=/rally/params.json',
               '--target-hosts=https://admin:xxx@es-client-service:9200']

and I got the same message in logs,

[root@k8master1 spark_helm]# kubectl logs esrally-644lr -n es -f

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[ERROR] Cannot race. not enough values to unpack (expected 2, got 1).

Getting further help:
*********************
* Check the log files in /rally/.rally/logs for errors.
* Read the documentation at https://esrally.readthedocs.io/en/2.2.0/.
* Ask a question on the forum at https://discuss.elastic.co/tags/c/elastic-stack/elasticsearch/rally.
* Raise an issue at https://github.com/elastic/rally/issues and include the log files in /rally/.rally/logs.

Please correct me If i am doing anything wrong.

Thanks,
Ganeshbabu R

@r.ganeshbabu I suspect it is that this:

...ca_certs=/data/certificates/cacert.pem"

should be this (colon instead of equals):

...ca_certs:/data/certificates/cacert.pem"

Yes @RickBoyd you are right..

I tried redeployed again and I got the below messages in logs,

[WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled.
[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[ERROR] Cannot race. Error in race control (Expected a git repository at [/rally/.rally/benchmarks/tracks/default] but the directory does not exist.)

Note:- I dont have internet access to this cluster.

To resolve the above issue I tried the below things,

  1. tried mounting the local path to pod using hostpath,
        volumeMounts:
        - name: esrally-data
          mountPath: /rally/.rally
        - name: trackparams-cfg-vol
          mountPath: /rally/params.json
          subPath: params.json
        - name: cert-ca
          mountPath: /data/certificates/
        - name: tracks
          mountPath: /rally/.rally/benchmarks
      restartPolicy: OnFailure
      volumes:
      - name: esrally-data
        persistentVolumeClaim:
          claimName: esrally-claim
      - name: trackparams-cfg-vol
        configMap:
          name: esrally-trackparams-cfg
      - name: cert-ca
        secret:
          secretName: elasticsearch-certs
      - name: tracks
        hostPath:
          path: /root/ganesh/prometheus-grafana/.rally/benchmarks

and I got this message in pod logs,

[root@k8master1 ~]# kubectl logs esrally-ct7bv -n es -f

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[ERROR] Cannot race. Error in race control ([/rally/.rally/benchmarks/tracks/default] must be a git repository.

Please run:
git -C /rally/.rally/benchmarks/tracks/default init)

Getting further help:
*********************
* Check the log files in /rally/.rally/logs for errors.
* Read the documentation at https://esrally.readthedocs.io/en/2.2.0/.
* Ask a question on the forum at https://discuss.elastic.co/tags/c/elastic-stack/elasticsearch/rally.
* Raise an issue at https://github.com/elastic/rally/issues and include the log files in /rally/.rally/logs.

can you please correct me if I am doing anything wrong.

Thanks,
Ganeshbabu R

Please see this guide in the Offline Usage section about tracks: Offline Usage — Rally 2.2.1 documentation

The track and corpora will have to be made available locally to the docker container in order to proceed

Hi @RickBoyd @Bradley_Deam

I have followed this article Offline Usage — Rally 2.2.1 documentation

below is the steps I have tried,

downloaded the script in server

# downloads the script from Github
curl -O https://raw.githubusercontent.com/elastic/rally-tracks/master/download.sh
chmod u+x download.sh
# download all data for the geonames track
./download.sh geonames

tar file also got created and i extracted it and files are stored in

/root/.rally/benchmarks

Extract with tar -xf rally-track-data-geonames.tar (will be extracted to ~/.rally/benchmarks).

Ex:-

[root@k8master1 default]# ll
total 8
-rw-r--r-- 1 root root 2896 Jun 23 10:29 README.md
-rwxr-xr-x 1 root root 2217 Jun 23 10:29 download.sh
drwxr-xr-x 2 root root   91 Jun 23 10:29 eql
drwxr-xr-x 4 root root  112 Jun 23 10:29 eventdata
drwxr-xr-x 4 root root  145 Jun 23 10:29 geonames
drwxr-xr-x 4 root root  112 Jun 23 10:29 geopoint
drwxr-xr-x 5 root root  126 Jun 23 10:29 geopointshape
drwxr-xr-x 5 root root  126 Jun 23 10:29 geoshape
drwxr-xr-x 5 root root  175 Jun 23 10:29 http_logs
drwxr-xr-x 4 root root   95 Jun 23 10:29 metricbeat
drwxr-xr-x 4 root root  180 Jun 23 10:29 nested
drwxr-xr-x 5 root root  126 Jun 23 10:29 noaa
drwxr-xr-x 5 root root  142 Jun 23 10:29 nyc_taxis
drwxr-xr-x 4 root root  112 Jun 23 10:29 percolator
drwxr-xr-x 4 root root  128 Jun 23 10:29 pmc
drwxr-xr-x 4 root root  144 Jun 23 10:29 so
[root@k8master1 default]# pwd
/root/.rally/benchmarks/tracks/default

and I tried the make the track path available locally to the pod using hostpath

        volumeMounts:
        - name: esrally-data
          mountPath: /rally/.rally
        - name: trackparams-cfg-vol
          mountPath: /rally/params.json
          subPath: params.json
        - name: cert-ca
          mountPath: /data/certificates/
        - name: tracks
          mountPath: /rally/.rally/benchmarks
      restartPolicy: OnFailure
      volumes:
      - name: esrally-data
        persistentVolumeClaim:
          claimName: esrally-claim
      - name: trackparams-cfg-vol
        configMap:
          name: esrally-trackparams-cfg
      - name: cert-ca
        secret:
          secretName: elasticsearch-certs
      - name: tracks
        hostPath:
          path: /root/.rally/benchmarks

and I got the below error in pod logs,

[root@k8master1 default]# kubectl logs esrally-tpwkh -n es -f

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.


[ERROR] Cannot race. Error in race control ([/root/.rally/benchmarks/tracks/default] must be a git repository.

Please run:
git -C /root/.rally/benchmarks/tracks/default init)

Getting further help:
*********************
* Check the log files in /root/.rally/logs for errors.
* Read the documentation at https://esrally.readthedocs.io/en/2.2.0/.
* Ask a question on the forum at https://discuss.elastic.co/tags/c/elastic-stack/elasticsearch/rally.
* Raise an issue at https://github.com/elastic/rally/issues and include the log files in /root/.rally/logs.

I have used offline mode in esrally race command and please help me to resolve this issue it would be really helpful.

Thanks,
Ganeshbabu R

Hi @RickBoyd

I am able to solve the above issues but I am getting the below errors,

[root@k8master1 ~]# kubectl logs esrally-4z5jt -n es -f

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[WARNING] Local changes in [/root/.rally/benchmarks/tracks/default] prevent tracks update from remote. Please commit your changes.
[WARNING] Local changes in [/root/.rally/benchmarks/tracks/default] prevent tracks update from remote. Please commit your changes.
[WARNING] indexing_total_time is 68 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 152 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] flush_total_time is 39 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[INFO] Racing on track [geonames], challenge [append-no-conflicts] and car ['external'] with version [7.10.2].


[ERROR] Cannot race. Error in load generator [0]
        Cannot run task [delete-index]: 'request_start'

Getting further help:
*********************
* Check the log files in /root/.rally/logs for errors.
* Read the documentation at https://esrally.readthedocs.io/en/2.2.0/.
* Ask a question on the forum at https://discuss.elastic.co/tags/c/elastic-stack/elasticsearch/rally.
* Raise an issue at https://github.com/elastic/rally/issues and include the log files in /root/.rally/logs.

attached the esrally manifest file,

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: esrally-claim
spec:
  storageClassName: gp2
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
kind: ConfigMap 
apiVersion: v1 
metadata:
  name: esrally-trackparams-cfg
data:
  params.json: |
    {
       "number_of_shards": 20,
       "number_of_replicas": 0,
       "ingest_percentage": 100,
       "bulk_size": 10000,
       "bulk_indexing_clients": 8,
       "index_settings": { "refresh_interval":-1 }
    }  
---
apiVersion: batch/v1
kind: Job
metadata:
  name: esrally
spec:
  template:
    metadata:
      name: esrally
    spec:
      securityContext:
        runAsUser: 0
      containers:
      - name: esrally
        image: elastic/rally:2.2.0
        imagePullPolicy: IfNotPresent
        args: ['race',
               '--track=geonames',
               '--test-mode',
               '--offline',
               '--pipeline=benchmark-only',
               '--client-options="basic_auth_user:admin,basic_auth_password:admin,timeout:120,use_ssl:false,verify_certs:false,ca_certs:/data/certificates/cacert.pem"',
               '--track-params=/rally/params.json',
               '--target-hosts=https://admin:xxxx@es-opendistro-es-client-service:9200']
        resources:
          requests:
            cpu: "2"
            memory: "8Gi"
          limits:
            cpu: "2"
            memory: "8Gi"
        volumeMounts:
        - name: esrally-data
          mountPath: /rally/.rally
        - name: trackparams-cfg-vol
          mountPath: /rally/params.json
          subPath: params.json
        - name: cert-ca
          mountPath: /data/certificates/
        - name: esrally
          mountPath: /root/.rally/benchmarks
      restartPolicy: OnFailure
      volumes:
      - name: esrally-data
        persistentVolumeClaim:
          claimName: esrally-claim
      - name: trackparams-cfg-vol
        configMap:
          name: esrally-trackparams-cfg
      - name: cert-ca
        secret:
          secretName: elasticsearch-certs
      - name: esrally
        hostPath:
          path: /root/.rally/benchmarks
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: rally
                operator: In
                values:
                - "true"
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - rally #change the value if there is a change in the app name
              topologyKey: kubernetes.io/hostname

pls let me know your thoughts how to solve this.

Thanks,
Ganeshbabu R

The next course of action is to attach a container or use some other method to browse this persistent volume claim, as the Rally log will be at<there>/logs/rally.log and see the full error message

Hi @RickBoyd

Checked the rally.log and found the below error,

2021-07-02 09:19:14,988 ActorAddr-(T|:40815)/PID:26 esrally.driver.driver INFO Creating iteration-count based schedule with [None] distribution for [delete-index] with [0] warmup iterations and [1] iterations.
2021-07-02 09:19:14,989 ActorAddr-(T|:40815)/PID:26 esrally.driver.driver INFO iteration-count-based schedule will determine when the schedule for [delete-index] terminates.
2021-07-02 09:19:14,906 ActorAddr-(T|:32827)/PID:23 esrally.driver.driver INFO [6/8] workers reached join point [0/33].
2021-07-02 09:19:14,992 ActorAddr-(T|:40815)/PID:26 esrally.driver.driver ERROR Could not execute schedule
Traceback (most recent call last):

  File "/usr/local/lib/python3.8/site-packages/esrally/driver/driver.py", line 1415, in __call__
    total_ops, total_ops_unit, request_meta_data = await execute_single(runner, self.es, params, self.on_error)

  File "/usr/local/lib/python3.8/site-packages/esrally/driver/driver.py", line 1484, in execute_single
    return_value = await runner(es, params)

  File "/usr/local/lib/python3.8/site-packages/esrally/driver/runner.py", line 267, in __call__
    return await self.delegate(*args)

  File "/usr/local/lib/python3.8/site-packages/esrally/driver/runner.py", line 382, in __call__
    return_value = await self.delegate(*args)

  File "/usr/local/lib/python3.8/site-packages/esrally/driver/runner.py", line 315, in __call__
    return await self.delegate(self.client_extractor(args[0]), *args[1:])

  File "/usr/local/lib/python3.8/site-packages/esrally/driver/runner.py", line 2398, in __call__
    return_value = await self.delegate(es, params)

  File "/usr/local/lib/python3.8/site-packages/esrally/driver/runner.py", line 1177, in __call__
    elif only_if_exists and await es.indices.exists(index=index_name):

  File "/usr/local/lib/python3.8/site-packages/elasticsearch/_async/client/indices.py", line 330, in exists
    return await self.transport.perform_request(

  File "/usr/local/lib/python3.8/site-packages/elasticsearch/_async/transport.py", line 287, in perform_request
    await self._async_call()

  File "/usr/local/lib/python3.8/site-packages/elasticsearch/_async/transport.py", line 132, in _async_call
    await self._async_init()

  File "/usr/local/lib/python3.8/site-packages/elasticsearch/_async/transport.py", line 117, in _async_init
    self.set_connections(self.hosts)

  File "/usr/local/lib/python3.8/site-packages/elasticsearch/transport.py", line 212, in set_connections
    connections = list(zip(connections, hosts))

  File "/usr/local/lib/python3.8/site-packages/elasticsearch/transport.py", line 208, in _create_connection
    return self.connection_class(**kwargs)

  File "/usr/local/lib/python3.8/site-packages/esrally/async_connection.py", line 164, in __init__
    super().__init__(host=host,

  File "/usr/local/lib/python3.8/site-packages/elasticsearch/_async/http_aiohttp.py", line 201, in __init__
    raise ImproperlyConfigured("ca_certs parameter is not a path")

elasticsearch.exceptions.ImproperlyConfigured: ca_certs parameter is not a path


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/usr/local/lib/python3.8/site-packages/esrally/driver/driver.py", line 1417, in __call__
    request_end = request_context.request_end

  File "/usr/local/lib/python3.8/site-packages/esrally/client.py", line 54, in __aexit__
    request_start = self.request_start

  File "/usr/local/lib/python3.8/site-packages/esrally/client.py", line 46, in request_start
    return self.ctx["request_start"]

KeyError: 'request_start'

the error was due to ca_cert didnt configure properly and resolved the issue by configured rightly.

Thanks for your guidance.

Ganeshbabu R

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.