Issue with Elasticsearch performance

Elastic Stack Elasticsearch
1

Hi,

When I try to query data from ES using Kibana, I am getting Gateway Timeout. I am using Nginx as a reverse proxy(Nginx, Kibana, ES coordinating node are installed on the same server and I have 3 Data nodes with 10 different indexes with 5 Primary and 1 Replica shards each).

I suspect issue would be at ES index level and not with Kibana or Nginx. Because all other indexes are working fine without any issues, only one index is timing out.

Some of the settings I use are;

    `elasticsearch.requestTimeout: 900000 in kibana.yml`

    -Xms16g
    -Xmx16g in all elasticsearch.yml (RAM is 32GB on each server)

In Nginx error file, the error I see is,

upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxx.xx.xx, server: kibanaserver, request: "POST /elasticsearch/_msearch HTTP/1.1", upstream: "http://kibanaserver:5601/elasticsearch/_msearch", host: "kibanaserver", referrer: "http://kibanaserver/app/kibana"

I don't see any errors in Kibana. Please let me know how to debug the issue and how to improve performance of ES.

Thanks

2

Does it happen without the reverse proxy?
Do you have Monitoring from X-Pack to see what is happening at the time of the timeouts?

3

I haven’t tried without reverse proxy because I can’t reach kibana without reverse proxy due to port issues.

Is there any way where I can directly query elasticsearch? Also am I missing any settings in elasticsearch? I feel like elasticsearch is utilizing more virtual memory on it’s server. Should I set mlockall to true or is it set to true by default? Please clarify..

4

Elasticsearch will only ever use heap memory.

The docs are your friends for these sorts of questions, if you can't find something let us know! Memory lock check | Elasticsearch Reference [6.2] | Elastic

5

By default NGiNX will timeout a connection in 60secs. You may want to look into the NGiNX config and make sure it is set to a little long than you Kibana timeouts.