When trying to index documents in elastic, the following exception is frequently getting raised:
ConnectionTimeout caused by -
ReadTimeoutError(HTTPSConnectionPool(host='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxeaf.us-west-2.aws.found.io', port='9243'):
Read timed out. (read timeout=10))
Looking at the logs in elasticsearch, I'm seeing this message:
[instance-0000000001] Received response for a request that has timed out, sent [22492ms] ago, timed out [0ms] ago, action [internal:coordination/fault_detection/leader_check], node [{instance-0000000000}{xxxxx}{xxxxx}{10.0.0.200}{10.0.0.200:19808}{dimrt}{logical_availability_zone=zone-0, server_name=instance-0000000000.xxxxxxeaf, availability_zone=us-west-2a, xpack.installed=true, data=hot, instance_configuration=aws.data.highio.i3, transform.node=true, region=us-west-2}], id [25109964]
A google search on the topic did not yield anything particularly useful. Many suggested to increase the timeout window from 10 seconds to 30 seconds, but this seemed to have no effect on the issue.
I'm really puzzled by these messages and I don't know why they are happening. Does anyone have any suggestions on what to do?
These are the specs of our elastic deployment:
Deployment version: v7.9.1
Performance Dashboard:
Instances:
Thanks for the help in advance!