Esrally Urllib3HttpConnection

Elastic Stack Elasticsearch
1

2020-04-06 00:59:42,445 -not-actor-/PID:15373 elasticsearch WARNING Connection <Urllib3HttpConnection: http://xxx.xxx.xx.xx:9200> has failed for 2 times in a row, putting on 120 second timeout.

2020-04-06 00:59:42,447 -not-actor-/PID:15375 elasticsearch WARNING POST http://xxx.xxx.xxx.xxx9200/_bulk [status:503 request:0.091s]

2020-04-06 00:59:42,466 -not-actor-/PID:15373 elasticsearch WARNING Undecodable raw error response from server: Expecting value: line 1 column 1 (char 0)

I get this in my rallylog, I don't know why.

2

Can you curl the Elasticsearch node from the rally host?

3

yes i can curl from rally host.... I dont if i am making a mistake in using the command

esrally --distribution-version=7.4.2 --track pmc --target-hosts=x.x.x.x9200,x.x.x.x:9200,x.x.x.x:9200 --pipeline=benchmark-only --client-options="use_ssl:true,basic_auth_user:'username',basic_auth_password:'password',verify_certs:false"

4

It seems like Rally tries to connect through HTTP even though it looks like you have a secured cluster. Can you try and change the connection string to include https:// and see if that makes any difference?

5

Since Elasticsearch is reporting 503 error, have you also checked in the Elasticsearch logs for clues?

6

I can see pmc index in my cluster, many time rally will fail at this point

.track.params INFO Will read [574199] lines from [/home/centos/.rally/benchmarks/data/pmc/documents.json] starting from line [0] with bulk size [1].

surprisingly, esrally will continue and will give some result but in the rally log i get timeout error.
in the elastic search i get ELASTICSEARCH.TRANSPORT.RemoteTransportException.

7

Please show the full error from the Elasticsearch logs.

8

Hi there is not much in elasticsearch log but rally is showing

elasticsearch WARNING POST
https://34.192.225.69:9200/_bulk [status:N/A request:10.010s]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib64/python3.6/ssl.py", line 968, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib64/python3.6/ssl.py", line 830, in read
return self._sslobj.read(len, buffer)
File "/usr/lib64/python3.6/ssl.py", line 587, in read
v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

9

The pmc track has very large docs and can cause stability issues to your cluster if it's underprovisioned, especially in terms of memory.

As suggested by @Christian_Dahlqvist there should be some useful entries in the Elasticsearch log, but you can also check a) if regular curl operations against ES work when Rally reports time outs b) memory/cpu/disk/network usage on the Elasticsearch and Rally nodes.

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.