Request Timeout after 30000 - Kibana process getting killed


(Ravi Shanker Reddy Kourla) #1

Hi All,

Sometimes Kibana shows a blank white page and I don;t see response.
My ES cluster health is fine and it is in green state.
Previously I used to restart the kibana and after restart I am able to user kibana without any issues.
Today again, kibana was showing blank white page, I though the same issue and I restarted the kibana.
After restart, my kibana process is hetting stopped after some time.
When I check the kibana logs I saw the below timeout message and due to this kibana process is getting stopped.

{"name":"Kibana","hostname":"xxxxx","pid":15207,"level":50,"err":{"message":"Request Timeout after 30000ms","name":"Error","stack":"Error: Request Timeout after 30000ms\n at null. (/home/release/release-independent/elk/kibana/kibana-4.1.1-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:282:15)\n at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)"},"msg":"","time":"2016-06-30T17:08:11.727Z","v":0}

I restarted kibana multiple times, but due to above issue kibana process is getting stopped.
To fix this issue, I restarted complete ES cluster today. After ES cluster kibana is working now.

Can any one please let me know what is the rrot cause for the timeout and why kibana sometimes shows the blan page?
Please help me out guys..

Thanks,
Ravi


(Jon Budzenski) #2

On the Kibana end, you can increase the timeout for elasticsearch requests by changing the request_timeout setting in your kibana.yml. Kibana is having a difficult time getting quick responses from elasticsearch, so it's possible your cluster is under heavy load or something in between is interfering.


(Ravi Shanker Reddy Kourla) #3

Thank you for the response.
As you mentioned, I analyzed the ES cluster and found that it was doing GC which is taking longer time.
We analyzed the heap dump and found the root cause, guess what, the issue is with filed data cache.
We limited the field data cache and used doc_values for all fields which are not analyzed and modified all visualizations in which we are using analyzed fields.
Now heap and GC in control.


#4

Hi!
I increased the request_timeout and I have the same problem.
My elasticsearch version is 2.3.4 and kibana version 4.5.3.
Elasticsearch and kibana are running in a debian server with 6gb RAM and 4 cores without ssd.
Elasticsearch has 355 indices (one per day).
Any solution? =S
Thanks in advance.


(system) #5