Index command raises timeout error

[MKH (Maryam Khosravi)]

I am new to elasticsearch and I have some problem when I try to index a document (a python dictionary) into elasticsearch. Could someone please help? I used this command to create and instance of Elasticsearch in my code:

es = Elasticsearch([self.host], http_auth=(self.user_name, self.psw), scheme=self.scheme)

Then I try to ping to my instance to make sure the connection is done and it always successful:

if es.ping:
    print('Connection is done successfully')

else:
    print('Connection failed')

My document is very small and it is just a small dictionary with 10 key:value items. I try to import this doc to es using index command :

doc = {...}
res = self.es.index(index=index_name, body=doc, request_timeout=30)

But I always get this error (I have deleted some of the error stack to make it short):

Traceback (most recent call last):
  File "/app/vbuild/RHEL7-x86_64/python/3.6-addons-pytest/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/app/vbuild/RHEL7-x86_64/python/3.6-addons-pytest/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/app/vbuild/RHEL7-x86_64/python/3.6.6/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/app/vbuild/RHEL7-x86_64/python/3.6.6/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/app/vbuild/RHEL7-x86_64/python/3.6.6/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/app/vbuild/RHEL7-x86_64/python/3.6.6/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/app/vbuild/RHEL7-x86_64/python/3.6.6/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/app/vbuild/RHEL7-x86_64/python/3.6.6/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/app/vbuild/RHEL7-x86_64/python/3.6.6/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 ...
  File "/app/vbuild/RHEL7-x86_64/python/3.6-addons-pytest/lib/python3.6/site-packages/urllib3/util/retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
 ...
  File "/app/vbuild/RHEL7-x86_64/python/3.6-addons-pytest/lib/python3.6/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='dafinfrantools-es.sero.wh.rnd.internal.ericsson.com', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 ...
packages/elasticsearch/connection/http_urllib3.py", line 261, in perform_request
    raise ConnectionTimeout("TIMEOUT", str(e), e)
elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPSConnectionPool(host='dafinfrantools-es.sero.wh.rnd.internal.ericsson.com', port=443): Read timed out. (read timeout=30))

More strange thing is that sometimes even with this error I can see my data is indexed in Kibana!
Could you please help?

[spinscale (Alexander Reelsen)]

Hey,

so a read timeout means, that the python client was waiting for data to be returned but that did not happen. First step is to make sure that the network connection between your application and the Elasticsearch endpoint is stable. Sometimes there are firewalls inbetween that drop connections after a certain amount of time.

I could imagine that the ping works flawless because it is not sending a lot of data, where as your index operations are indeed sending more data and take more time to come back, but this is just guessing.

Also, if the timeout only occurs, when you are waiting for a response, then sending the request to Elasticsearch might just have succeeded and that explains why you saw data.

So before doing anything else, I would try to check if there are issues with the network connection...

Hope this helps as a start!

--Alex