Running benchmark-only stalls

Elastic Stack Elasticsearch
1

I'm running esrally with the benchmark-only pipeline, but it's stalling at Watch your steps!, nothing happens.

esrally --pipeline=benchmark-only --target-hosts=https://[server].com --track=http_logs

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/


************************************************************************
************** WARNING: A dark dungeon lies ahead of you  **************
************************************************************************

Rally does not have control over the configuration of the benchmarked
Elasticsearch cluster.

Be aware that results may be misleading due to problems with the setup.
Rally is also not able to gather lots of metrics at all (like CPU usage
of the benchmarked cluster) or may even produce misleading metrics (like
the index size).

************************************************************************
****** Use this pipeline only if you are aware of the tradeoffs.  ******
*************************** Watch your step! ***************************
************************************************************************

I've opened ports, I've tried running from a mac and a linux machine, I'm using esrally==1.4.1. I've waited for over an hour, the process keeps running but nothing happens.

Any ideas on what might be wrong or how to debug this?

2

Is there anything in the Rally logs?

3

Ah, yes, lots of errors. Ports are open so I don't know why it wouldn't connect. I'll have to look into the connection errors...

2020-05-01 08:30:43,327 -not-actor-/PID:6588 elasticsearch WARNING GET https://[server]:443/_cluster/health?wait_for_nodes=%3E%3D1 [status:N/A request:31.524s]
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 238, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 301, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fdd4428f978>: Failed to establish a new connection: [Errno 110] Connection timed out
4

Do you have a proxy in place as you are connecting to port 443? Can you curl to the cluster from the node where you are running Rally?

5

It was a site-wide firewall in action that prevented the connection. Got it sorted, thank you.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.