Trying to load data using Rally into Elastic Search

sarba · April 12, 2018, 8:27pm

Trying to load data using rally into Elastic Search and facing this issue:

root@esdb01-cqs01:~/.rally/benchmarks/tracks/default/contactdata# esrally --track=contactdata --target-hosts=esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200 --track-params="bulk_size:2000" --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true,basic_auth_user:'elastic',basic_auth_password:''"

____        ____

/ __ ____ / / / __
/ // / __ `/ / / / / /
/ , / // / / / // /
// ||_,///_, /
/____/

[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[INFO] Writing logs to /root/.rally/logs/rally_out_20180412T201810Z.log

************** WARNING: A dark dungeon lies ahead of you **************

Rally does not have control over the configuration of the benchmarked
Elasticsearch cluster.

Be aware that results may be misleading due to problems with the setup.
Rally is also not able to gather lots of metrics at all (like CPU usage
of the benchmarked cluster) or may even produce misleading metrics (like
the index size).

****** Use this pipeline only if you are aware of the tradeoffs. ******
*************************** Watch your step! ***************************

[WARNING] Could not terminate all internal processes within timeout. Please check and force-terminate all Rally processes.
[ERROR] Cannot race. Got an unexpected result during benchmarking: [Poison<<esrally.racecontrol.Setup object at 0x7f7d9886c908>>: Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 850, in validate_conn
conn.connect()
File "/usr/local/lib/python3.4/dist-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.4/dist-packages/urllib3/util/ssl.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.4/ssl.py", line 365, in wrap_socket
_context=self)
File "/usr/lib/python3.4/ssl.py", line 601, in init
self.do_handshake()
File "/usr/lib/python3.4/ssl.py", line 828, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:600)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/elasticsearch/connection/http_urllib3.py", line 147, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
File "/usr/local/lib/python3.4/dist-packages/esrally/client.py", line 47, in urlopen
return self.pool.urlopen(method, url, body=body, retries=retries, headers=headers, **kw)
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.4/dist-packages/urllib3/util/retry.py", line 333, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.4/dist-packages/urllib3/packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 850, in validate_conn
conn.connect()
File "/usr/local/lib/python3.4/dist-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.4/dist-packages/urllib3/util/ssl.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.4/ssl.py", line 365, in wrap_socket
_context=self)
File "/usr/lib/python3.4/ssl.py", line 601, in init
self.do_handshake()
File "/usr/lib/python3.4/ssl.py", line 828, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:600)

sarba · April 12, 2018, 8:47pm

I am using Elastic Search 6.2.2 with X-pack

dliappis · April 13, 2018, 7:27am

Hello @sarba,

The client options you are using instruct Rally to use https (SSL/TLS):

 --client-options="use_ssl:true,verify_certs:true

however, the error message you are receiving indicates that Rally failed trying to use the encrypted http protocol with TLS/SSL to talk to Elasticsearch at the provided port.

One possible reason is that Elasticsearch hasn't been properly configured to use SSL for the http client.
This is easy to check using something like:

curl https://esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200

which will return an error if it's unable to establish the encrypted connection, such as:

curl: (35) gnutls_handshake() failed: An unexpected TLS packet was received.

(one alternative method of checking whether Elasticsearch is accepting TLS encrypted http communication is with openssl s_client -showcerts -connect esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200; it will exit showing "no peer certificate available" if SSL/TLS isn't supported).

sarba · April 13, 2018, 2:51pm

Thanks for your reply.

root@esdb01-cqs01:~/.rally/logs# curl -u elastic:'' -XGET 'http://esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200'
{
"name" : "esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com",
"cluster_name" : "elastic.cqs01.stg.stage",
"cluster_uuid" : "zOvq3bVkQfeCB_OHAf9vMQ",
"version" : {
"number" : "6.2.2",
"build_hash" : "10b1edd",
"build_date" : "2018-02-16T19:01:30.685723Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
root@esdb01-cqs01:~/.rally/logs# curl -u elastic:'' -XGET 'https://esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200'
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
root@esdb01-cqs01:~/.rally/logs#

root@esdb01-cqs01:~/.rally/logs# openssl s_client -showcerts -connect esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200
CONNECTED(00000003)
139774469691040:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:795:

no peer certificate available

No client certificate CA names sent

SSL handshake has read 7 bytes and written 295 bytes

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE

root@esdb01-cqs01:~/.rally/logs#

But Elastic search nodes are using secure connection and I have certificates and x-pack is installed.

sarba · April 13, 2018, 3:47pm

I have enabled Https. Now it shows this error:

root@esdb01-cqs01:/etc/elasticsearch# curl https://esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

curl -u elastic:'' https://esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200 -k
{
"name" : "esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com",
"cluster_name" : "elastic.cqs01.stg.stage",
"cluster_uuid" : "zOvq3bVkQfeCB_OHAf9vMQ",
"version" : {
"number" : "6.2.2",
"build_hash" : "10b1edd",
"build_date" : "2018-02-16T19:01:30.685723Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

dliappis · April 14, 2018, 12:58pm

Hey @sarba,

Now that you've properly enabled SSL (with self signed certificates, it appears) you should be able to execute Rally with a minor change in your arguments:

verify_certs:true --> verify_certs:false

sarba · April 16, 2018, 5:41pm

Yes, it is working when I specify verify_certs:false, but failed when specify : verify_certs:true.

system · May 14, 2018, 5:41pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Rally data load fails 80/20 all the time Elasticsearch rally	6	1001	May 18, 2018
Rally does not have control over the configuration of the benchmarked Elasticsearch cluster Elasticsearch rally	4	1056	March 11, 2020
Rally gets stuck while benchmarking Elasticsearch rally	5	1435	October 25, 2018
Cannot race, worker has exited prematurely Elasticsearch rally	6	126	July 1, 2024
Rally errors Elasticsearch rally	11	3224	July 5, 2017

Trying to load data using Rally into Elastic Search

root@esdb01-cqs01:~/.rally/logs# openssl s_client -showcerts -connect esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200 CONNECTED(00000003) 139774469691040:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:795:

no peer certificate available

No client certificate CA names sent

SSL handshake has read 7 bytes and written 295 bytes

New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE

Related topics

root@esdb01-cqs01:~/.rally/logs# openssl s_client -showcerts -connect esdb01-cqs01.db.us-west-1a.stg1.ebs.ebcolo.com:9200
CONNECTED(00000003)
139774469691040:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:795:

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE