When I try to query data from ES using Kibana, I am getting Gateway Timeout. I am using Nginx as a reverse proxy(Nginx, Kibana, ES coordinating node are installed on the same server and I have 3 Data nodes with 10 different indexes with 5 Primary and 1 Replica shards each).
I suspect issue would be at ES index level and not with Kibana or Nginx. Because all other indexes are working fine without any issues, only one index is timing out.
Some of the settings I use are;
`elasticsearch.requestTimeout: 900000 in kibana.yml` -Xms16g -Xmx16g in all elasticsearch.yml (RAM is 32GB on each server)
In Nginx error file, the error I see is,
upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxx.xx.xx, server: kibanaserver, request: "POST /elasticsearch/_msearch HTTP/1.1", upstream: "http://kibanaserver:5601/elasticsearch/_msearch", host: "kibanaserver", referrer: "http://kibanaserver/app/kibana"
I don't see any errors in Kibana. Please let me know how to debug the issue and how to improve performance of ES.