# ESRally issue when running EventData track

**URL:** <https://discuss.elastic.co/t/esrally-issue-when-running-eventdata-track/300858>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [March 28, 2022, 2:48pm UTC](https://discuss.elastic.co/t/esrally-issue-when-running-eventdata-track/300858 "2022-03-28T14:48:39Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kailas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailas/32/55376_2.png) [@Kailas](https://discuss.elastic.co/u/Kailas)\
**Post date:** [March 28, 2022, 2:48pm UTC](https://discuss.elastic.co/t/esrally-issue-when-running-eventdata-track/300858/1 "2022-03-28T14:48:39Z")

</div>

There were some network changes and the Rally process stopped in between and the state of ES was in “yellow/red”. Hence I deleted to index from the ES cluster.

Now, when I run the below Rally command

```auto
esrally race --kill-running-processes --track=eventdata --target-hosts=172.29.183.11:9200,172.29.183.12:9200,172.29.183.13:9200 --pipeline=benchmark-only --track-repository=eventdata --track-params="number_of_shards:3,number_of_replicas:1,bulk_size:10000,bulk_indexing_clients:15,daily_logging_volume:100GB,starting_point:2022-01-01,number_of_days:3,bulk_indexing_reqs_per_sec:15,search_clients:10" --challenge=index-fixed-load-and-query”  

```

I get the error

```auto
Running check-cluster-health [100% done]
Running fieldstats [0% done]
[ERROR] Cannot race. Error in load generator [0]
        Cannot run task [fieldstats]: No matching data found for field '@timestamp' in pattern 'elasticlogs-*'.

```

“

Is there a way in Rally to force the data to download and re-index so the test runs successfully.

Thanks  
Kailas

---

<div class="post-metadata">

**Author:** ![RickBoyd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rickboyd/32/80022_2.png) [@RickBoyd](https://discuss.elastic.co/u/RickBoyd)\
**Post date:** [March 28, 2022, 3:15pm UTC](https://discuss.elastic.co/t/esrally-issue-when-running-eventdata-track/300858/2 "2022-03-28T15:15:54Z")

</div>

Hi @Kailas,

Thanks for using Rally! Please note from the eventdata repository ([here](https://github.com/elastic/rally-eventdata-track#index-and-query-logs-fixed-daily-volume)) for the challenge you're executing:

> Requires executing `index-logs-fixed-daily-volume` first.

Please give that a shot to initialize the index templates and index data prior to the execution of the `index-fixed-load-and-query` challenge.

---

<div class="post-metadata">

**Author:** ![Kailas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailas/32/55376_2.png) [@Kailas](https://discuss.elastic.co/u/Kailas)\
**Post date:** [March 29, 2022, 2:43am UTC](https://discuss.elastic.co/t/esrally-issue-when-running-eventdata-track/300858/3 "2022-03-29T02:43:46Z")

</div>

Thanks for the solution. However, a quick question when the load-and-query reaches to 89-90% I always get

```auto
Running index-fixed-throughput,content_issues-dashboard-25%,discover-30... [89% done]
[WARNING] Could not terminate all internal processes within timeout. Please check and force-terminate all Rally processes.
[ERROR] Cannot race. Error in load generator [0]
       Cannot run task [index-fixed-throughput]: Request returned an error. Error type: transport, Description: Cannot connect to host 172.29.183.13:9200 ssl:default [Connect call failed ('172.29.183.13', 9200)] (Cannot connect to host 172.29.183.13:9200 ssl:default [Connect call failed ('172.29.183.13', 9200)])

```

Any solution for this one ?

Thanks  
Kailas

---

<div class="post-metadata">

**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\
**Post date:** [March 29, 2022, 8:45am UTC](https://discuss.elastic.co/t/esrally-issue-when-running-eventdata-track/300858/4 "2022-03-29T08:45:17Z")

</div>

Please take a look at `~/.rally/logs/rally.log` and also at your Elasticsearch logs and whether Elasticsearch is still reachable. I suspect that the high load is affecting Elasticsearch's performance.

Note that the default `timeout` value for the client that Rally uses is `60s`. See the [docs](https://esrally.readthedocs.io/en/stable/command_line_reference.html?highlight=timeout#client-options) on how to increase it via `--client-options`.

If you haven't already read it, please read this section of the docs about [latency vs service\_time](https://esrally.readthedocs.io/en/stable/faq.html#what-do-latency-and-service-time-mean-and-how-do-they-relate-to-the-took-field-that-elasticsearch-returns) that should help you analyze results.

---

<div class="post-metadata">

**Author:** ![Kailas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailas/32/55376_2.png) [@Kailas](https://discuss.elastic.co/u/Kailas)\
**Post date:** [March 30, 2022, 2:54am UTC](https://discuss.elastic.co/t/esrally-issue-when-running-eventdata-track/300858/5 "2022-03-30T02:54:48Z")

</div>

Hi,  
Thanks for the suggesstion. I increased the timeout to 120 but still the same issue persists.  
From the Rally logs

```auto
 2022-03-29 06:59:35,339 -not-actor-/PID:18593 elasticsearch WARNING POST http://172.29.183.13:9200/_bulk [status:N/A request:1.234s]
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/elasticsearch/_async/http_aiohttp.py", line 291, in perform_request
    async with self.session.request(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 559, in _request
    await resp.start(conn)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read() # type: ignore[union-attr]
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer
2022-03-29 06:59:35,341 -not-actor-/PID:18592 elasticsearch WARNING POST http://172.29.183.13:9200/_bulk [status:N/A request:2.862s]
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/elasticsearch/_async/http_aiohttp.py", line 291, in perform_request
    async with self.session.request(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 559, in _request
    await resp.start(conn)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read() # type: ignore[union-attr]
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected
2022-03-29 06:59:35,341 -not-actor-/PID:18596 elasticsearch WARNING POST http://172.29.183.13:9200/_bulk [status:N/A request:2.944s]
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/elasticsearch/_async/http_aiohttp.py", line 291, in perform_request
    async with self.session.request(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 559, in _request
    await resp.start(conn)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read() # type: ignore[union-attr]
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected
2022-03-29 06:59:35,341 -not-actor-/PID:18595 elasticsearch WARNING POST http://172.29.183.13:9200/_bulk [status:N/A request:2.869s]
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/elasticsearch/_async/http_aiohttp.py", line 291, in perform_request
    async with self.session.request(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 559, in _request
    await resp.start(conn)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read() # type: ignore[union-attr]
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected
2022-03-29 06:59:35,341 -not-actor-/PID:18594 elasticsearch WARNING POST http://172.29.183.13:9200/_bulk [status:N/A request:3.001s]
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/elasticsearch/_async/http_aiohttp.py", line 291, in perform_request
    async with self.session.request(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 559, in _request
    await resp.start(conn)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read() # type: ignore[union-attr]
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected
2022-03-29 06:59:35,345 -not-actor-/PID:18593 elasticsearch WARNING Connection <AIOHttpConnection: http://172.29.183.13:9200> has failed for 1 times in a row, putting on 60 second timeout.
2022-03-29 06:59:35,347 -not-actor-/PID:18592 elasticsearch WARNING Connection <AIOHttpConnection: http://172.29.183.13:9200> has failed for 1 times in a row, putting on 60 second timeout.
2022-03-29 06:59:35,349 -not-actor-/PID:18594 elasticsearch WARNING Connection <AIOHttpConnection: http://172.29.183.13:9200> has failed for 1 times in a row, putting on 60 second timeout.
2022-03-29 06:59:35,350 -not-actor-/PID:18596 elasticsearch WARNING Connection <AIOHttpConnection: http://172.29.183.13:9200> has failed for 1 times in a row, putting on 60 second timeout.
2022-03-29 06:59:35,352 -not-actor-/PID:18595 elasticsearch WARNING Connection <AIOHttpConnection: http://172.29.183.13:9200> has failed for 1 times in a row, putting on 60 second timeout.
2022-03-29 06:59:35,352 -not-actor-/PID:18593 elasticsearch WARNING POST http://172.29.183.13:9200/_bulk [status:N/A request:2.650s]
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/elasticsearch/_async/http_aiohttp.py", line 291, in perform_request
    async with self.session.request(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 559, in _request
    await resp.start(conn)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read() # type: ignore[union-attr]
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected
2022-03-29 06:59:35,352 -not-actor-/PID:18592 elasticsearch WARNING POST http://172.29.183.13:9200/_bulk [status:N/A request:1.735s]
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/elasticsearch/_async/http_aiohttp.py", line 291, in perform_request
    async with self.session.request(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 559, in _request
    await resp.start(conn)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read() # type: ignore[union-attr]
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer
2022-03-29 06:59:35,354 -not-actor-/PID:18594 elasticsearch WARNING POST http://172.29.183.13:9200/_bulk [status:N/A request:2.491s]
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/elasticsearch/_async/http_aiohttp.py", line 291, in perform_request
    async with self.session.request(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 559, in _request
    await resp.start(conn)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read() # type: ignore[union-attr]
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected
2022-03-29 07:00:01,100 -not-actor-/PID:18546 elasticsearch WARNING GET http://172.29.183.13:9200/_cluster/state/master_node [status:N/A request:0.001s]
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 251, in perform_request
    response = self.pool.urlopen(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/urllib3/util/retry.py", line 525, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/http/client.py", line 1007, in _send_output
    self.send(msg)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/http/client.py", line 947, in send
    self.connect()
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fae1fde6af0>: Failed to establish a new connection: [Errno 111] Connection refused
2022-03-29 07:00:01,100 -not-actor-/PID:18546 elasticsearch WARNING Connection <Urllib3HttpConnection: http://172.29.183.13:9200> has failed for 1 times in a row, putting on 60 second timeout.
2022-03-29 07:00:02,914 -not-actor-/PID:18600 elasticsearch WARNING POST http://172.29.183.11:9200/_msearch?pre_filter_shard_size=1 [status:N/A request:0.001s]
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/connector.py", line 986, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs) # type: ignore[return-value] # noqa
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
    raise exceptions[0]
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
    sock = await self._connect_sock(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
    await self.sock_connect(sock, address)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/selector_events.py", line 496, in sock_connect
    return await fut
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/selector_events.py", line 528, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('172.29.183.11', 9200)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/elasticsearch/_async/http_aiohttp.py", line 291, in perform_request
    async with self.session.request(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/connector.py", line 907, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
    raise last_exc
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/connector.py", line 992, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 172.29.183.11:9200 ssl:default [Connect call failed ('172.29.183.11', 9200)]
2022-03-29 07:00:02,916 -not-actor-/PID:18600 elasticsearch WARNING Connection <AIOHttpConnection: http://172.29.183.11:9200> has failed for 1 times in a row, putting on 60 second timeout.
2022-03-29 07:00:14,752 -not-actor-/PID:18603 elasticsearch WARNING POST http://172.29.183.11:9200/_msearch?pre_filter_shard_size=1 [status:N/A request:0.001s]
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/connector.py", line 986, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs) # type: ignore[return-value] # noqa
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
    raise exceptions[0]
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
    sock = await self._connect_sock(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
    await self.sock_connect(sock, address)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/selector_events.py", line 496, in sock_connect
    return await fut
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/selector_events.py", line 528, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('172.29.183.11', 9200)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/elasticsearch/_async/http_aiohttp.py", line 291, in perform_request
    async with self.session.request(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/connector.py", line 907, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
    raise last_exc
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/connector.py", line 992, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 172.29.183.11:9200 ssl:default [Connect call failed ('172.29.183.11', 9200)]
2022-03-29 07:00:14,754 -not-actor-/PID:18603 elasticsearch WARNING Connection <AIOHttpConnection: http://172.29.183.11:9200> has failed for 1 times in a row, putting on 60 second timeout.
2022-03-29 07:00:14,755 -not-actor-/PID:18603 elasticsearch WARNING POST http://172.29.183.13:9200/_msearch?pre_filter_shard_size=1 [status:N/A request:0.001s]
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/site-packages/aiohttp/connector.py", line 986, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs) # type: ignore[return-value] # noqa
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
    raise exceptions[0]
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
    sock = await self._connect_sock(
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
    await self.sock_connect(sock, address)
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/selector_events.py", line 496, in sock_connect
    return await fut
  File "/root/.pyenv/versions/3.8.10/lib/python3.8/asyncio/selector_events.py", line 528, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('172.29.183.13', 9200)

```

Thanks  
Kailas

---

<div class="post-metadata">

**Author:** ![Kailas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailas/32/55376_2.png) [@Kailas](https://discuss.elastic.co/u/Kailas)\
**Post date:** [March 30, 2022, 6:43am UTC](https://discuss.elastic.co/t/esrally-issue-when-running-eventdata-track/300858/6 "2022-03-30T06:43:33Z")

</div>

Ok, on looking into more details - The entire elastic cluster fails as it shows "outofmemory" error

```auto
Mar 29 22:31:24 e1-node1 systemd[1]: Starting Elasticsearch...
Mar 29 22:31:42 e1-node1 systemd[1]: Started Elasticsearch.
Mar 30 02:23:57 e1-node1 systemd-entrypoint[1353]: [13951.559s][warning][gc,alloc] elasticsearch[node-1][[elasticlogs-2022-01-03][1]: Lucene Merge Thread #809]: Retried w...61459 words
Mar 30 02:23:57 e1-node1 systemd-entrypoint[1353]: java.lang.OutOfMemoryError: Java heap space
Mar 30 02:23:57 e1-node1 systemd-entrypoint[1353]: Dumping heap to /var/lib/elasticsearch/java_pid1353.hprof ...
Mar 30 02:23:57 e1-node1 systemd-entrypoint[1353]: Unable to create /var/lib/elasticsearch/java_pid1353.hprof: File exists
Mar 30 02:23:57 e1-node1 systemd-entrypoint[1353]: Terminating due to java.lang.OutOfMemoryError: Java heap space
Mar 30 02:23:57 e1-node1 systemd[1]: elasticsearch.service: main process exited, code=exited, status=3/NOTIMPLEMENTED
Mar 30 02:23:57 e1-node1 systemd[1]: Unit elasticsearch.service entered failed state.
Mar 30 02:23:57 e1-node1 systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

```

```auto
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default_local]
        ... 39 more
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of coordinating operation [coordinating_and_primary_bytes=67823416, replica_bytes=40280217, all_bytes=108103633, coordinating_operation_bytes=101757, max_coordinating_and_primary_bytes=107374182]
        at org.elasticsearch.index.IndexingPressure.markCoordinatingOperationStarted(IndexingPressure.java:76) ~[elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:186) ~[elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:91) ~[elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) ~[elasticsearch-7.16.1.jar:7.16.1]
        ... 34 more
[root@e1-node1 elasticsearch]# tail -f es_cluster1.log
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default_local]
        ... 39 more
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of coordinating operation [coordinating_and_primary_bytes=67823416, replica_bytes=40280217, all_bytes=108103633, coordinating_operation_bytes=101757, max_coordinating_and_primary_bytes=107374182]
        at org.elasticsearch.index.IndexingPressure.markCoordinatingOperationStarted(IndexingPressure.java:76) ~[elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:186) ~[elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:91) ~[elasticsearch-7.16.1.jar:7.16.1]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) ~[elasticsearch-7.16.1.jar:7.16.1]
        ... 34 more

```

I have data nodes with RAM of 64 GB. I haven't changed any settings. What should I do in this case ?

Thanks  
Kailas

---

<div class="post-metadata">

**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\
**Post date:** [March 30, 2022, 8:35am UTC](https://discuss.elastic.co/t/esrally-issue-when-running-eventdata-track/300858/7 "2022-03-30T08:35:18Z")

</div>

> [@Kailas](#):
>
> I have data nodes with RAM of 64 GB. I haven't changed any settings. What should I do in this case ?

Is this the first time you are running this workload (seems to based on the [first comment](https://discuss.elastic.co/t/esrally-issue-when-running-eventdata-track/300858)?

If yes, and given that Elasticsearch runs out of heap memory, you either need to reduce the load that benchmark generates and/or validate that your heap settings are correct.

But more importantly, I think you should consider what are you actually trying to achieve; what is the purpose of your benchmark? This will guide you in selecting the right parameters. I strongly recommend that you watch Daniel Mitterdorfer's presentation on the [7 deadly sins of benchmarking](https://www.elastic.co/elasticon/conf/2018/sf/the-seven-deadly-sins-of-elasticsearch-benchmarking), slides [here](https://daniel.mitterdorfer.name/talks/2018/Seven%20Deadly%20Sins%20of%20Elasticsearch%20Benchmarking.pdf).

---

<div class="post-metadata">

**Author:** ![Kailas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailas/32/55376_2.png) [@Kailas](https://discuss.elastic.co/u/Kailas)\
**Post date:** [March 30, 2022, 1:56pm UTC](https://discuss.elastic.co/t/esrally-issue-when-running-eventdata-track/300858/8 "2022-03-30T13:56:29Z")

</div>

Many thanks, changing the heap to 30g worked.

Regards  
Kailas

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 27, 2022, 1:57pm UTC](https://discuss.elastic.co/t/esrally-issue-when-running-eventdata-track/300858/9 "2022-04-27T13:57:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.