Benchmarking using ESRally

Elastic Stack Elasticsearch
1

Hey there,
I'm new to ESRally and I would appreciate some assistance in running esrally with ES running on a remote server.

The set-up:
There is a remote server which I connect to it via ssh.. it has a single elasticsearch pod running in it. So I have the ip address of the server and even that of pod. The elasticsearch in the pod already has some indices and data in it. Due to some reasons Im not able to install esrally on that server..
So how can i connect to the server and connect to the pod and run create-track command from my local machine so that I can get a custom track and then run some benchmarking on the ES running in a pod. If there are some tutorials or hands-on about this then please point it out I will follow that.

Thank you.

2

Hello,

Would the offline install (Installation - Rally 2.7.1 documentation) work for you? Or is there a way to copy the data somewhere an install of Rally could reach it?

The last option could be to set up an SSH tunnel (What is an SSH Tunnel & SSH Tunneling?) which I think could work. But we cannot help you directly with that, sorry.

3

Thanks @Quentic_Pradet for the reply.
So if I install esrally offline on the remote server. Should I have to install esrally in the pods also to carry out benchmarking since the Elasticsearch is running inside pods ?

4

Hi Dershan,

No, you don't need to install esrally inside the pods. You can use --pipeline=benchmark-only option to benchmark existing K8s cluster. The only requirement is esrally needs to be able to connect to TCP/9200 or any other port where Elasticsearch running in the pod is exposed. See here for more details: Tips and Tricks - Rally 2.7.1 documentation .

Please note testing an existing cluster may affect other traffic in this cluster, so it's not recommended to benchmark a production cluster unless you know what you're doing :slight_smile:

Kind regards.

1 Like
5

Thanks @gbanasiak for your reply and no I'm not benchmarking a production cluster, its for testing purpose only, thanks for your help :slight_smile: .

6

Hello,
I used offline method to install ESrally on a server. In that server I have one Elasticsearch statefulset pod running. I was able to connect to the pod and do any operations using curl commands using the pod ip with 9200 port number. I ran this command esrally create-track --track=acme --target-hosts=<pod_ip>:9200 --indices="topology-devices-,audit_trail_2023_5" --output-path=~/tracks with the same pod ip and port number used in curl commands. But Im getting the following error
[ERROR] Cannot create-track. The client noticed that the server is not Elasticsearch and we do not support this unknown product.. I can assure that the ip address belongs to a elasticsearch pod. I'm not sure why it says so... Is it something related to elasticsearch version, should I have to mention it somewhere to rally which version of ES I'm using ? I would really appreciate some assistance in this matter.

7

Can you share GET / response from your cluster retrieved via curl including the exact command used? Please add -i option to include the headers returned by Elasticsearch.

Example based on Elastic Cloud deployment (hence HTTPS and port 9243):

% curl -i -u elastic:<password> https://<URL>:9243
HTTP/2 200
content-type: application/json
x-cloud-request-id: <REDACTED>
x-elastic-product: Elasticsearch
x-found-handling-cluster: <REDACTED>
x-found-handling-instance: instance-0000000000
content-length: 564
date: Wed, 14 Jun 2023 08:16:32 GMT

{
  "name" : "instance-0000000000",
  "cluster_name" : "<REDACTED>",
  "cluster_uuid" : "<REDACTED>",
  "version" : {
    "number" : "8.8.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "c01029875a091076ed42cdb3a41c10b1a9a5a20f",
    "build_date" : "2023-05-23T17:16:07.179039820Z",
    "build_snapshot" : false,
    "lucene_version" : "9.6.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Please also share Rally stack trace from ~/.rally/logs/rally.log corresponding to the error you mentioned. GET / response seems to be failing this verification but let's double-check.

8

For the command curl -i <ipaddress>:9200
Output:

content-type: application/json; charset=UTF-8
content-length: 345

{
  "name" : "elasticsearch-1",
  "cluster_name" : "search-es-cluster-main",
  "cluster_uuid" : "OzTzf1q6Q_epkB5Z7hiq7g",
  "version" : {
    "number" : "5.6.16",
    "build_hash" : "3a740d1",
    "build_date" : "2019-03-13T15:33:36.565Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

I was using curl -X GET /_cat/indices

curl -X GET "<ipaddress>:9200/_cat/indices?v=true&pretty" -i

Output:
HTTP/1.1 200 OK
content-type: text/plain; charset=UTF-8
content-length: 21593
<and all indices list>

Contents of log file when I ran esrally create-track

2023-06-14 09:40:17,571 -not-actor-/PID:20774 esrally.rally INFO OS [uname_result(system='Linux', node='***-156-248.***.com', release='4.15.0-1039-fips', version='#44+recert2+tpmfix $
2023-06-14 09:40:17,571 -not-actor-/PID:20774 esrally.rally INFO Python [namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50856176, name='cpython', version=sys.ve$
2023-06-14 09:40:17,571 -not-actor-/PID:20774 esrally.rally INFO Rally version [2.8.0]
2023-06-14 09:40:17,571 -not-actor-/PID:20774 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2023-06-14 09:40:17,571 -not-actor-/PID:20774 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2023-06-14 09:40:17,571 -not-actor-/PID:20774 esrally.rally INFO Cleaning track dependency directory [/home/core/.rally/libs]...
2023-06-14 09:40:17,571 -not-actor-/PID:20774 esrally.client.factory INFO Creating ES client connected to [{'host': '<ipaddress>', 'port': 9200}] with options [{'timeout': 60}]
2023-06-14 09:40:17,631 -not-actor-/PID:20774 esrally.rally ERROR A fatal error occurred while running subcommand [create-track].
Traceback (most recent call last):
  File "/home/core/venv/lib/python3.8/site-packages/esrally/rally.py", line 1186, in dispatch_sub_command
    tracker.create_track(cfg)
  File "/home/core/venv/lib/python3.8/site-packages/esrally/tracker/tracker.py", line 85, in create_track
    info = client.info()
  File "/home/core/venv/lib/python3.8/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)
  File "/home/core/venv/lib/python3.8/site-packages/elasticsearch/_sync/client/__init__.py", line 2296, in info
    return self.perform_request(  # type: ignore[return-value]
  File "/home/core/venv/lib/python3.8/site-packages/esrally/client/synchronous.py", line 170, in perform_request
    _ProductChecker.raise_error(self._verified_elasticsearch, info_meta, info_body)
  File "/home/core/venv/lib/python3.8/site-packages/esrally/client/synchronous.py", line 74, in raise_error
    raise UnsupportedProductError(message, meta=meta, body=body)
elasticsearch.UnsupportedProductError: The client noticed that the server is not Elasticsearch and we do not support this unknown product

I hope the info is enough, if anything is missing pls highlight

9

Thank you for sharing.

You're using very old Elasticsearch version 5.6.16 which is not supported by Rally, see Elasticsearch Version Support - Rally 2.8.0 documentation.

10

Thanks for taking your time and helping out.
Is there any lower version of rally that supports ES of v5.

11

Official support for Elasticsearch 5.x was dropped in this PR, in Rally version 2.2.0. Formally you would need to use Rally 2.1.0 or older.

The recommended option is to upgrade Elasticsearch cluster to supported version, and use most recent Rally version.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.