Rally test failed with SSL error

Elastic Stack Elasticsearch
1

Installed rally today to setup benchmarking test.

but it fails with some kind of SSL certificate error. I do not have any security setup on my elasticsearch. but going out from this machine to internet need http proxy which I setup.

here is the some part of the log
2019-05-17 19:37:49,340 -not-actor-/PID:14504 esrally.utils.net INFO Connecting via proxy URL [http://104.129.204.32:80] to the Internet (picked up from the env variable [http_proxy]).
2019-05-17 19:37:50,284 -not-actor-/PID:14504 esrally.rally INFO Detected a working Internet connection.

It detected proxy properly.
First some kind of error in log is
2019-05-17 19:37:54,464 ActorAddr-(T|:46529)/PID:14513 esrally.actor INFO Received a benchmark failure from [ActorAddr-(T|:39139)] and will forward it now.

and then following

2019-05-17 19:37:54,467 -not-actor-/PID:14504 esrally.racecontrol ERROR A benchmark failure has occurred
2019-05-17 19:37:54,467 -not-actor-/PID:14504 esrally.racecontrol INFO Telling benchmark actor to exit.
2019-05-17 19:37:51,874 ActorAddr-(T|:45923)/PID:14631 esrally.utils.repo INFO Rebasing on [7] in [/root/.rally/benchmarks/teams/default] for distribution version [7.0.1].
2019-05-17 19:37:54,468 -not-actor-/PID:14504 esrally.rally INFO Attempting to shutdown internal actor system.
2019-05-17 19:37:51,951 ActorAddr-(T|:45923)/PID:14631 esrally.mechanic.java_resolver INFO Allowed JDK versions are [12, 11, 8].
2019-05-17 19:37:54,468 ActorAddr-(T|:46529)/PID:14513 esrally.actor INFO BenchmarkActor received unknown message [ActorExitRequest] (ignoring).
2019-05-17 19:37:52,207 ActorAddr-(T|:45923)/PID:14631 esrally.mechanic.java_resolver INFO Detected JDK with major version [8] in [/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64/jre].
2019-05-17 19:37:54,469 ActorAddr-(T|:39139)/PID:14572 esrally.actor INFO MechanicActor#receiveMessage unrecognized(msg = [<class 'thespian.actors.ActorExitRequest'>] sender = [ActorAddr-(T|:46529)])
2019-05-17 19:37:52,208 ActorAddr-(T|:45923)/PID:14631 esrally.mechanic.supplier INFO Resolved download URL [https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-7.0.1-linux-x86_64.tar.gz] for version [7.0.1]
2019-05-17 19:37:54,471 ActorAddr-(T|:46529)/PID:14513 esrally.actor INFO BenchmarkActor received unknown message [ActorExitRequest] (ignoring).
2019-05-17 19:37:54,471 ActorAddr-(T|:37001)/PID:14630 esrally.actor INFO mechanic.Dispatcher#receiveMessage unrecognized(msg = [<class 'thespian.actors.ActorExitRequest'>] sender = [ActorAddr-(T|:39139)])
2019-05-17 19:37:52,209 ActorAddr-(T|:45923)/PID:14631 esrally.mechanic.supplier INFO Starting download of Elasticsearch [7.0.1]
2019-05-17 19:37:54,456 ActorAddr-(T|:45923)/PID:14631 esrally.actor ERROR Cannot process message [<esrally.mechanic.mechanic.StartNodes object at 0x7f41e74c31d0>]
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 595, in urlopen
    self._prepare_proxy(conn)

  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 816, in _prepare_proxy
    conn.connect()

  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)

  File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)

  File "/usr/lib64/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)

  File "/usr/lib64/python3.6/ssl.py", line 814, in __init__
    self.do_handshake()

  File "/usr/lib64/python3.6/ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()

  File "/usr/lib64/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:841)

This is test machine setup with elk 7.01. can I just not use that rather then setting up new elk.

2

Hello @elasticforme,

You didn't provide the exact Rally command, but I presume you didn't specify a pipeline. By default Rally will attempt to setup Elasticsearch for you but in your case you have an externally provisioned cluster so please refer to the documents for pipelines. You'll need to use the pipeline benchmark-only: https://esrally.readthedocs.io/en/stable/pipelines.html?highlight=pipeline#benchmark-only

1 Like
3

This is command I used.
I will go through this document and test the way it has describe.

esrally --distribution-version=7.0.1 --track=metricbeat --challenge=append-no-conflicts

4

Allright it is working.
this is what I did. if someone else has same problem.
Thank you @dliappis

/root/.local/bin/esrally list tracks
download.sh geoshape

mv that to /root and untar it.

/root/.local/bin/esrally --pipeline=benchmark-only --target-hosts=localhost:9200

and it started working.

then I downloaded other tracks as well.
now I should be able to benchmark using different track as biggest track is geoshape.

5

@elasticforme your command looks good to me and specifying distribution-version will automatically set the right pipeline, so this is a different problem to what I was thinking. It works fine locally for me.

Can you check the Elasticsearch logs? Assuming you've just executed the above rally command, logs are under ~/.rally/benchmarks/races/<date>/rally-node-0/logs/server/*.log so you can use something like this to get the logs and paste here: cat $(ls -td -- ~/.rally/benchmarks/races/* | head -n1)/rally-node-0/logs/server/rally-benchmark.log

6

Ok glad to hear you have things working. Your earlier command esrally --distribution-version=7.0.1 --track=metricbeat --challenge=append-no-conflicts will attempt to download run Elasticsearch 7.0.1 by Rally itself (it will listen on port 39200).

Using the latest command you provided it will assume an Elasticsearch is already running on localhost:9200.

7

Run few test on local system and on remote elk clusters.

are there premade dashboard jason out there for it?

I stumble upon this thread and it has lot of dashboard. but can't figure out how to configure it.

https://elasticsearch-benchmark-analytics.elastic.co/app/kibana#/dashboards?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-90d,mode:quick,to:now))&filter=

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.