When I try to query data from ES using Kibana, I am getting Gateway Timeout. I am using Nginx as a reverse proxy(Nginx, Kibana, ES coordinating node are installed on the same server and I have 3 Data nodes with 10 different indexes with 5 Primary and 1 Replica shards each).
I suspect issue would be at ES index level and not with Kibana or Nginx. Because all other indexes are working fine without any issues, only one index is timing out.
Some of the settings I use are;
`elasticsearch.requestTimeout: 900000 in kibana.yml`
-Xms16g
-Xmx16g in all elasticsearch.yml (RAM is 32GB on each server)
In Nginx error file, the error I see is,
upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxx.xx.xx, server: kibanaserver, request: "POST /elasticsearch/_msearch HTTP/1.1", upstream: "http://kibanaserver:5601/elasticsearch/_msearch", host: "kibanaserver", referrer: "http://kibanaserver/app/kibana"
I don't see any errors in Kibana. Please let me know how to debug the issue and how to improve performance of ES.
I haven’t tried without reverse proxy because I can’t reach kibana without reverse proxy due to port issues.
Is there any way where I can directly query elasticsearch? Also am I missing any settings in elasticsearch? I feel like elasticsearch is utilizing more virtual memory on it’s server. Should I set mlockall to true or is it set to true by default? Please clarify..
By default NGiNX will timeout a connection in 60secs. You may want to look into the NGiNX config and make sure it is set to a little long than you Kibana timeouts.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.