I run kibana + elasticsearch in a small production environment, and I get this error "Request Timeout after 30000ms" every few query run.
I'm opening a thread here in elasticsearch forum because my guess is that the problem is more an elasticsearch issue than a kibana issue:
from my investigations, elasticsearch randomly stops to reply even to simple requests like this
curl -XGET http://localhost:9200/_cluster/health?pretty , that hangs for ever.
- I open a terminal on node2
- after a few requests like curl -XGET http://localhost:9200/_cluster/health?pretty , curl hangs for ever
- I open another terminal on the same node and run again the curl request while the first curl still running: sometime I get a reply, sometime curls hangs
I found the same behaviour on all 3 nodes.
servers: ubuntu 14.04 running on Amazon EC2 m3.large instances (2 CPUs, 7,5 GB RAM)
kibana runs on a dedicated server, while elasticsearch runs on 3 other servers.
Each node has 6GB allocated for the heap using -Xms6g -Xmx6g
here the stats for the clusters nodes
> $ curl -XGET 'localhost:9200/_cat/nodes?v' ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 172.31.11.125 22 99 7 0.37 0.32 0.32 mdi - node2.tr.elasticsearch.local 172.31.14.208 58 98 16 0.50 0.33 0.32 mdi * node3.tr.elasticsearch.local 172.31.14.207 10 99 1 0.17 0.20 0.25 mdi - node4.tr.elasticsearch.local
kibana is configured to use node2, using elasticsearch.url: "http://172.31.11.125:9200" in kibana.yml
ping time between nodes is around 0.3 ms
> $ ping 172.31.14.208
PING 172.31.14.208 (172.31.14.208) 56(84) bytes of data.
64 bytes from 172.31.14.208: icmp_seq=1 ttl=64 time=0.325 ms
64 bytes from 172.31.14.208: icmp_seq=2 ttl=64 time=0.323 ms
64 bytes from 172.31.14.208: icmp_seq=3 ttl=64 time=0.327 ms
elasticsearch.log file don't say nothing.
No cpu spikes are shown when the curl hangs.
Any advice to what to further investigate will be appreciated.