Elasticsearch server is rejecting any communication after certain amount of time interval if there is no activity from source server and connection is getting lost.? Can connection be open at all times, Is there any timeout setting available to avoid this..?
Do you have a reverse proxy or a firewall or anything like that between the client and elasticsearch nodes?
Yeah we have a firewall between Client and Elasticsearch nodes, which is open and connection is working, but we have this issue intermittently when there is no activity for sometime. Elasticsearch is closing all client connections and causing timeouts at the client end for any other query after timeout.
I think you have to configure your proxy/firewall for this. I'm pretty sure people also had issues in the past with their firewall. Did you google a bit or search on this site?
I checked with our network team they said once the connection is successful there is nothing wrong with network, most of the google suggestions are conflicting. Some of them says its something to do with elasticsearch, and some are like to increase the timeout at java elastic client. What's more strange is we are having this issue only on higher environments.
Elasticsearch does not have an idle connection timeout (by default anyway) - it will let HTTP connections sit idle indefinitely. However, firewalls almost always do time out idle connections, and is likely the cause of your problems. That's not to say there's anything "wrong" with the network if so, it's behaving as configured, but the configuration is not what Elasticsearch needs.
Elasticsearch also sends TCP keepalives on incoming HTTP connections to try and prevent them from being closed in this way. This doesn't always work, particularly if the firewall is configured to close connections before the first keepalive is sent. If that's what is happening, you need to adjust either your firewall or your keepalive config.
Thanks.. keepalive config is it an elasticsearch configuration setting.?
No, it's a sysctl.