Trying to load data using Rally into Elastic Search

Elastic Stack Elasticsearch
1

Trying to load data using rally into Elastic Search and facing this issue:

root@esdb01-cqs01:~/.rally/benchmarks/tracks/default/contactdata# esrally --track=contactdata --target-hosts=esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200 --track-params="bulk_size:2000" --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true,basic_auth_user:'elastic',basic_auth_password:''"

____        ____

/ __ ____ / / / __
/ // / __ `/ / / / / /
/ , / // / / / // /
// ||_,///_, /
/____/

[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[INFO] Writing logs to /root/.rally/logs/rally_out_20180412T201810Z.log

************** WARNING: A dark dungeon lies ahead of you **************

Rally does not have control over the configuration of the benchmarked
Elasticsearch cluster.

Be aware that results may be misleading due to problems with the setup.
Rally is also not able to gather lots of metrics at all (like CPU usage
of the benchmarked cluster) or may even produce misleading metrics (like
the index size).

****** Use this pipeline only if you are aware of the tradeoffs. ******
*************************** Watch your step! ***************************

[WARNING] Could not terminate all internal processes within timeout. Please check and force-terminate all Rally processes.
[ERROR] Cannot race. Got an unexpected result during benchmarking: [Poison<<esrally.racecontrol.Setup object at 0x7f7d9886c908>>: Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 850, in validate_conn
conn.connect()
File "/usr/local/lib/python3.4/dist-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.4/dist-packages/urllib3/util/ssl.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.4/ssl.py", line 365, in wrap_socket
_context=self)
File "/usr/lib/python3.4/ssl.py", line 601, in init
self.do_handshake()
File "/usr/lib/python3.4/ssl.py", line 828, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:600)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/elasticsearch/connection/http_urllib3.py", line 147, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
File "/usr/local/lib/python3.4/dist-packages/esrally/client.py", line 47, in urlopen
return self.pool.urlopen(method, url, body=body, retries=retries, headers=headers, **kw)
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.4/dist-packages/urllib3/util/retry.py", line 333, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.4/dist-packages/urllib3/packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 850, in validate_conn
conn.connect()
File "/usr/local/lib/python3.4/dist-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.4/dist-packages/urllib3/util/ssl.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.4/ssl.py", line 365, in wrap_socket
_context=self)
File "/usr/lib/python3.4/ssl.py", line 601, in init
self.do_handshake()
File "/usr/lib/python3.4/ssl.py", line 828, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:600)

2

I am using Elastic Search 6.2.2 with X-pack

3

Hello @sarba,

The client options you are using instruct Rally to use https (SSL/TLS):

 --client-options="use_ssl:true,verify_certs:true

however, the error message you are receiving indicates that Rally failed trying to use the encrypted http protocol with TLS/SSL to talk to Elasticsearch at the provided port.

One possible reason is that Elasticsearch hasn't been properly configured to use SSL for the http client.
This is easy to check using something like:

curl https://esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200

which will return an error if it's unable to establish the encrypted connection, such as:

curl: (35) gnutls_handshake() failed: An unexpected TLS packet was received.

(one alternative method of checking whether Elasticsearch is accepting TLS encrypted http communication is with openssl s_client -showcerts -connect esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200; it will exit showing "no peer certificate available" if SSL/TLS isn't supported).

1 Like
4

Thanks for your reply.

root@esdb01-cqs01:~/.rally/logs# curl -u elastic:'' -XGET 'http://esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200'
{
"name" : "esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com",
"cluster_name" : "elastic.cqs01.stg.stage",
"cluster_uuid" : "zOvq3bVkQfeCB_OHAf9vMQ",
"version" : {
"number" : "6.2.2",
"build_hash" : "10b1edd",
"build_date" : "2018-02-16T19:01:30.685723Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
root@esdb01-cqs01:~/.rally/logs# curl -u elastic:'' -XGET 'https://esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200'
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
root@esdb01-cqs01:~/.rally/logs#

root@esdb01-cqs01:~/.rally/logs# openssl s_client -showcerts -connect esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200
CONNECTED(00000003)
139774469691040:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:795:

no peer certificate available

No client certificate CA names sent

SSL handshake has read 7 bytes and written 295 bytes

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE

root@esdb01-cqs01:~/.rally/logs#

But Elastic search nodes are using secure connection and I have certificates and x-pack is installed.

5

I have enabled Https. Now it shows this error:

root@esdb01-cqs01:/etc/elasticsearch# curl https://esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

curl -u elastic:'' https://esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200 -k
{
"name" : "esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com",
"cluster_name" : "elastic.cqs01.stg.stage",
"cluster_uuid" : "zOvq3bVkQfeCB_OHAf9vMQ",
"version" : {
"number" : "6.2.2",
"build_hash" : "10b1edd",
"build_date" : "2018-02-16T19:01:30.685723Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

6

Hey @sarba,

Now that you've properly enabled SSL (with self signed certificates, it appears) you should be able to execute Rally with a minor change in your arguments:

verify_certs:true --> verify_certs:false

7

Yes, it is working when I specify verify_certs:false, but failed when specify : verify_certs:true.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.