Index status as red with 1 unassigned shard

[Alp1]

Hi ,

I am running few rally tracks on our dev environment and i observed whenever i run pmc / nyc_taxis tracks the index goes into red state after some 5GB of data with 1 unassigned shard error. These tracks have data size > 20GB.

I am not sure why the index goes in to red state with 1 unassigned shard after 5 GB of data .Logs shows the timeout error. Could you give me some pointers

log shows below error -

urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='..,', port=8089): Read timed out. (read timeout=10)
2018-02-07 05:03:59,959 PID:14010 elasticsearch WARNING POST http://...:8089/_bulk [status:N/A request:10.043s]
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
six.raise_from(e, None)
File "", line 2, in raise_from
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.4/http/client.py", line 1208, in getresponse
response.begin()
File "/usr/lib/python3.4/http/client.py", line 380, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.4/http/client.py", line 342, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.4/socket.py", line 374, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/elasticsearch/connection/http_urllib3.py", line 147, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
File "/usr/local/lib/python3.4/dist-packages/esrally/client.py", line 47, in urlopen
return self.pool.urlopen(method, url, body=body, retries=retries, headers=headers, **kw)
File "/usr/local/lib/python3.4/dist-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.4/dist-packages/urllib3/util/retry.py", line 333, in increment

[danielmitterdorfer]

You can use the cluster allocation explain API to find out why you have unassigned shards.

The Rally error that you are posting is different though:

urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='..,', port=8089): Read timed out. (read timeout=10)

This means that the HTTP request has timed out. The timeout value is 10 seconds and you can set --client-options="timeout:60" to increase it to e.g. 60 seconds.

[Alp1]

Thanks Daniel.
Cluster allocation explain API is really helpful."Node Left" was the reason for unassigned shard error.

[danielmitterdorfer]

Perfect. Glad that it helped.