Kibana dashboard timeout 30000 and hangs


#1

Hello,

I am new to ELK Stack but it is really irritating that Kibana just does not want to work correctly. All the time when i try to visualize results even from 30 minutes of time frame, dashboard will fail to do so and if that was not enough after it times out whole kibana interface just cease to work it is not possible to do anything and only stack restart helps.

I have dockerized elk-stack with xpack installed and basic x-pack license. For figuring this out I did make 1 node dockerized elk stack on machine with parameters as below:
CPU i7
RAM 16gb
256gb ssd drive
Shards total size: ~5gb
Number of shards: 29

Status of node is yellow because there is 1 node and it is not possible to make replicas at least I assume this is the issue also unassigned shard are mainly elk system indicies.

"epoch": "1528378067",
"timestamp": "13:27:47",
"cluster": "elk-stack",
"status": "yellow",
"node.total": "1",
"node.data": "1",
"shards": "29",
"pri": "29",
"relo": "0",
"init": "0",
"unassign": "22",
"pending_tasks": "0",
"max_task_wait_time": "-",
"active_shards_percent": "56.9%"
In logs I get this error:

kibana_1         | {"type":"error","@timestamp":"2018-06-07T13:21:16Z","tags":["warning","monitoring-ui","kibana-monitoring"],"pid":1,"level":"error","error":{"message":"Request Timeout after 30000ms","name":"Error","stack":"Error: Request Timeout after 30000ms\n    at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:342:15\n    at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:371:7)\n    at ontimeout (timers.js:386:11)\n    at tryOnTimeout (timers.js:250:5)\n    at Timer.listOnTimeout (timers.js:214:5)"},"message":"Request Timeout after 30000ms"}
kibana_1         | {"type":"log","@timestamp":"2018-06-07T13:21:16Z","tags":["warning","monitoring-ui","kibana-monitoring"],"pid":1,"message":"Unable to fetch data from kibana collector"}
kibana_1         | Unhandled rejection TypeError: Cannot read property 'result' of undefined
kibana_1         |     at /usr/share/kibana/plugins/x-pack/plugins/monitoring/server/kibana_monitoring/lib/type_collector.js:91:50
kibana_1         |     at Array.filter (native)
kibana_1         |     at /usr/share/kibana/plugins/x-pack/plugins/monitoring/server/kibana_monitoring/lib/type_collector.js:91:29
kibana_1         |     at next (native)
kibana_1         |     at step (/usr/share/kibana/plugins/x-pack/plugins/monitoring/server/kibana_monitoring/lib/type_collector.js:18:202)
kibana_1         |     at /usr/share/kibana/plugins/x-pack/plugins/monitoring/server/kibana_monitoring/lib/type_collector.js:18:383
kibana_1         |     at bound (domain.js:280:14)
kibana_1         |     at runBound (domain.js:293:12)
kibana_1         |     at tryCatcher (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/util.js:11:23)
kibana_1         |     at Promise._settlePromiseFromHandler (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:489:31)
kibana_1         |     at Promise._settlePromise (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:546:18)
kibana_1         |     at Promise._settlePromise0 (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:591:10)
kibana_1         |     at Promise._settlePromises (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:674:18)
kibana_1         |     at Promise._fulfill (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:615:18)
kibana_1         |     at MappingPromiseArray.PromiseArray._resolve (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise_array.js:125:19)
kibana_1         |     at MappingPromiseArray._promiseFulfilled (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/map.js:97:18)
kibana_1         |     at Promise._settlePromise (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:551:26)
kibana_1         |     at Promise._settlePromise0 (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:591:10)
kibana_1         |     at Promise._settlePromises (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:674:18)
kibana_1         |     at Promise._fulfill (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:615:18)
kibana_1         |     at Promise._resolveCallback (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:416:57)
kibana_1         |     at Promise._settlePromiseFromHandler (/usr/share/kibana/plugins/x-pack/node_modules/bluebird/js/release/promise.js:502:17)

I was thinking that maybe it is heap issue but I have it set to 50% of machine RAM approx 8gb out of 16gb
I did try to change timeout to 60000ms in single node system it did not help at all, kibana just wait a bit longer but results in the same error. In multi node system it did not help and also did result in other issues and errors.


(Chris Roberson) #2

Hi @MrFrost,

In multi node system it did not help and also did result in other issues and errors.

What other issues/errors did you see?


#3

In multi node I also got errors:
Error 503 Service Unavailable: [search_phase_execution_exception] all shards failed:
Socket hang up (like a lot of this)


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.