One of the kibana node not loading

Elastic Stack Kibana
1

I have a ES cluster with 5 data nodes, 2 http nodes, 3 master nodes. On the 3 master nodes I have installed kibana, filebeat, metricbeat. However, the kibana takes a lot of time to load and finally gives error 'Request Timeout after 90000ms: Check the Elasticsearch Monitoring cluster network connection or the load level of the nodes.'

The ES cluster is a new cluster and currently it has no data. It has the default .kibana index, metricbeat and filebeat index

We have enabled filebeat system, elasticsearch, kibana module for filebeat and metricbeat system, kibana, kibana-xpack, elasticsearch,elasticsearch-xpack modules on the 3 kibana machines. I troublehsooted the issue with local port forwarding in the .ssh/config file and one of the kibana node out of 3 machines is causing the issue. 2 of the 3 machines work fine at localhost:5061. Kibana loads immediately with 2 machines, however, with the 3rd node it gives the error. I increased the timeout value still no success

Error log from Kibana

Mar 23 07:51:15 ex51ssd115 kibana[143084]: {"type":"error","@timestamp":"2020-03-23T07:49:45Z","tags":[],"pid":143084,"level":"error","error":{"message":"Request Timeout after 90000ms","name":"Error","stack":"Error: Request Timeout after 90000ms\n    at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:397:9\n    at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:429:7)\n    at ontimeout (timers.js:436:11)\n    at tryOnTimeout (timers.js:300:5)\n    at listOnTimeout (timers.js:263:5)\n    at Timer.processTimers (timers.js:223:10)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/api/monitoring/v1/clusters/fpQB8sKdTF-1aq_F3itD6w","path":"/api/monitoring/v1/clusters/fpQB8sKdTF-1aq_F3itD6w","href":"/api/monitoring/v1/clusters/fpQB8sKdTF-1aq_F3itD6w"},"message":"Request Timeout after 90000ms"}
2

Hi @priyashree,

Are you sure that all the configuration and firewall setting in the failing Kibana Machine are the same of all the other machines?
Can you post also the rest of the Kibana Logs before the error to see if there is something else that can be useful to debug the issue
It will be also useful to specify the verbose settings in the kibana.yml file to true to diagnose the issue better

logging.verbose: true
4

configuration and firewall settings are same

Mar 23 11:31:37 ex51ssd115 kibana[143084]: {"type":"error","@timestamp":"2020-03-23T11:31:37Z","tags":["error","monitoring"],"pid":143084,"level":"error","error":{"message":"Request Timeout after 90000ms","name":"Error","stack":"Error: Request Timeout after 90000ms\n    at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:397:9\n    at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:429:7)\n    at ontimeout (timers.js:436:11)\n    at tryOnTimeout (timers.js:300:5)\n    at listOnTimeout (timers.js:263:5)\n    at Timer.processTimers (timers.js:223:10)"},"message":"Request Timeout after 90000ms"}
Mar 23 11:31:37 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:28:36Z","tags":[],"pid":143084,"method":"post","statusCode":200,"req":{"url":"/api/monitoring/v1/clusters/fpQB8sKdTF-1aq_F3itD6w","method":"post","headers":{"x-forwarded-for":"zzz.zzz.zzz.zzz, ttt.ttt.ttt.ttt","host":"kibanabe.xxx.com","x-forwarded-proto":"http","x-real-ip":"ttt.ttt.ttt.ttt","connection":"close","content-length":"101","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","content-type":"application/json;charset=utf-8","kbn-version":"7.5.1","origin":"https://kibanabe.xxx.com","referer":"https://kibanabe.xxx.com/app/monitoring"},"remoteAddress":"xxx.xxx.xxx.xxx","userAgent":"xxx.xxx.xxx.xxx","referer":"https://kibanabe.xxx.com/app/monitoring"},"res":{"statusCode":200,"responseTime":181480,"contentLength":9},"message":"POST /api/monitoring/v1/clusters/fpQB8sKdTF-1aq_F3itD6w 200 181480ms - 9.0B"}
Mar 23 11:31:39 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:31:39Z","tags":["api"],"pid":143084,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":200,"responseTime":38,"contentLength":9},"message":"GET /api/status 200 38ms - 9.0B"}
Mar 23 11:31:58 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:31:56Z","tags":[],"pid":143084,"method":"post","statusCode":200,"req":{"url":"/api/monitoring/v1/clusters","method":"post","headers":{"x-forwarded-for":"zzz.zzz.zzz.zzz, ttt.ttt.ttt.ttt","host":"kibanabe.xxx.com","x-forwarded-proto":"http","x-real-ip":"xxx.xxx.xxx.xxx","connection":"close","content-length":"105","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","content-type":"application/json;charset=utf-8","kbn-version":"7.5.1","origin":"https://kibanabe.xxx.com","referer":"https://kibanabe.xxx.com/app/monitoring"},"remoteAddress":"xxx.xxx.xxx.xxx","userAgent":"xxx.xxx.xxx.xxx","referer":"https://kibanabe.xxx.com/app/monitoring"},"res":{"statusCode":200,"responseTime":1425,"contentLength":9},"message":"POST /api/monitoring/v1/clusters 200 1425ms - 9.0B"}
Mar 23 11:31:59 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:30:29Z","tags":["api"],"pid":143084,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":200,"responseTime":90005,"contentLength":9},"message":"GET /api/status 200 90005ms - 9.0B"}
Mar 23 11:31:59 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:31:59Z","tags":["api"],"pid":143084,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":200,"responseTime":18,"contentLength":9},"message":"GET /api/status 200 18ms - 9.0B"}
Mar 23 11:32:09 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:32:09Z","tags":["api"],"pid":143084,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":200,"responseTime":20,"contentLength":9},"message":"GET /api/status 200 20ms - 9.0B"}
Mar 23 11:32:19 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:32:19Z","tags":["api"],"pid":143084,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":200,"responseTime":15,"contentLength":9},"message":"GET /api/status 200 15ms - 9.0B"}
Mar 23 11:32:29 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:32:29Z","tags":["api"],"pid":143084,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":200,"responseTime":37,"contentLength":9},"message":"GET /api/status 200 37ms - 9.0B"}
Mar 23 11:32:32 ex51ssd115 kibana[143084]: {"type":"error","@timestamp":"2020-03-23T11:32:32Z","tags":["error","monitoring"],"pid":143084,"level":"error","error":{"message":"Request Timeout after 90000ms","name":"Error","stack":"Error: Request Timeout after 90000ms\n    at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:397:9\n    at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:429:7)\n    at ontimeout (timers.js:436:11)\n    at tryOnTimeout (timers.js:300:5)\n    at listOnTimeout (timers.js:263:5)\n    at Timer.processTimers (timers.js:223:10)"},"message":"Request Timeout after 90000ms"}
Mar 23 11:32:33 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:32:33Z","tags":[],"pid":143084,"method":"post","statusCode":200,"req":{"url":"/api/monitoring/v1/clusters","method":"post","headers":{"x-forwarded-for":"zzz.zzz.zzz.zzz, ttt.ttt.ttt.ttt","host":"kibanabe.xxx.com","x-forwarded-proto":"http","x-real-ip":"ttt.ttt.ttt.ttt","connection":"close","content-length":"105","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","content-type":"application/json;charset=utf-8","kbn-version":"7.5.1","origin":"https://kibanabe.xxx.com","referer":"https://kibanabe.xxx.com/app/monitoring"},"remoteAddress":"xxx.xxx.xxx.xxx","userAgent":"xxx.xxx.xxx.xxx","referer":"https://kibanabe.xxx.com/app/monitoring"},"res":{"statusCode":200,"responseTime":41,"contentLength":9},"message":"POST /api/monitoring/v1/clusters 200 41ms - 9.0B"}
Mar 23 11:32:39 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:32:39Z","tags":["api"],"pid":143084,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":200,"responseTime":19,"contentLength":9},"message":"GET /api/status 200 19ms - 9.0B"}
Mar 23 11:32:42 ex51ssd115 kibana[143084]: {"type":"log","@timestamp":"2020-03-23T11:32:42Z","tags":["license","warning","xpack"],"pid":143084,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: Request Timeout after 90000ms"}
Mar 23 11:32:42 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:29:42Z","tags":[],"pid":143084,"method":"post","statusCode":200,"req":{"url":"/api/monitoring/v1/clusters","method":"post","headers":{"x-forwarded-for":"zzz.zzz.zzz.zzz, ttt.ttt.ttt.ttt","host":"kibanabe.xxx.com","x-forwarded-proto":"http","x-real-ip":"ttt.ttt.ttt.ttt","connection":"close","content-length":"105","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","content-type":"application/json;charset=utf-8","kbn-version":"7.5.1","origin":"https://kibanabe.xxx.com","referer":"https://kibanabe.xxx.com/app/monitoring"},"remoteAddress":"xxx.xxx.xxx.xxx","userAgent":"xxx.xxx.xxx.xxx","referer":"https://kibanabe.xxx.com/app/monitoring"},"res":{"statusCode":200,"responseTime":180016,"contentLength":9},"message":"POST /api/monitoring/v1/clusters 200 180016ms - 9.0B"}
Mar 23 11:32:46 ex51ssd115 kibana[143084]: {"type":"log","@timestamp":"2020-03-23T11:32:46Z","tags":["license","info","xpack"],"pid":143084,"message":"Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active"}
Mar 23 11:32:46 ex51ssd115 kibana[143084]: {"type":"log","@timestamp":"2020-03-23T11:32:46Z","tags":["info","monitoring","kibana-monitoring"],"pid":143084,"message":"Starting monitoring stats collection"}
Mar 23 11:32:49 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:32:49Z","tags":["api"],"pid":143084,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":200,"responseTime":20,"contentLength":9},"message":"GET /api/status 200 20ms - 9.0B"}
Mar 23 11:32:55 ex51ssd115 kibana[143084]: {"type":"log","@timestamp":"2020-03-23T11:32:55Z","tags":["warning","monitoring","kibana-monitoring"],"pid":143084,"message":"Error: Request Timeout after 90000ms\n    at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:397:9\n    at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:429:7)\n    at ontimeout (timers.js:436:11)\n    at tryOnTimeout (timers.js:300:5)\n    at listOnTimeout (timers.js:263:5)\n    at Timer.processTimers (timers.js:223:10)"}
Mar 23 11:32:55 ex51ssd115 kibana[143084]: {"type":"log","@timestamp":"2020-03-23T11:32:55Z","tags":["warning","monitoring","kibana-monitoring"],"pid":143084,"message":"Unable to bulk upload the stats payload to the local cluster"}
Mar 23 11:32:59 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:32:59Z","tags":["api"],"pid":143084,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":200,"responseTime":36,"contentLength":9},"message":"GET /api/status 200 36ms - 9.0B"}
Mar 23 11:33:09 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:33:09Z","tags":["api"],"pid":143084,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":200,"responseTime":15,"contentLength":9},"message":"GET /api/status 200 15ms - 9.0B"}
Mar 23 11:33:19 ex51ssd115 kibana[143084]: {"type":"response","@timestamp":"2020-03-23T11:31:49Z","tags":["api"],"pid":143084,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":200,"responseTime":90005,"contentLength":9},"message":"GET /api/status 200 90005ms - 9.0B"}
Mar
5

Thanks for the log,
Do you see anything in the Elasticsearch logs that can be linked to this timeout?

What is the output of the cluster stats API ?

6

No error related to timeout in Elasticsearch logs. Also, the timeout error that I mentioned earlier appears comes at random times and sometimes kibana takes long to load.Also, the kibana max response time is shown as 1505 ms when clicked on 'Slack Monitoring'

If needed ES logs

[2020-03-23T08:55:29,214][WARN ][r.suppressed             ] [ex51ssd115] path: /_security/_authenticate, params: {}
org.elasticsearch.ElasticsearchException: Security must be explicitly enabled when using a [basic] license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node.
	at org.elasticsearch.xpack.security.rest.action.SecurityBaseRestHandler.checkFeatureAvailable(SecurityBaseRestHandler.java:75) ~[x-pack-security-7.5.1.jar:7.5.1]
	at org.elasticsearch.xpack.security.rest.action.SecurityBaseRestHandler.prepareRequest(SecurityBaseRestHandler.java:50) [x-pack-security-7.5.1.jar:7.5.1]
	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:87) [elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:69) [x-pack-security-7.5.1.jar:7.5.1]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:222) [elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:295) [elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166) [elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:322) [elasticsearch-7.5.1.jar:7.5.1]
7

Probably as written in the ES log exception it could be a matter of missing the security.enabled setting in the elasticsearch.yml file
Please check ES nodes yml file to see if the security flag is enabled

8

enabling xpack.security gives me the error to enable xpack.security.transport.ssl.enabled.We are using the basic version of Elasticsearch. We have 2 other cluster with the same settings as this cluster and those 2 are working fine without any issues. The issue is experienced only on this cluster. Please help

9

I'm not sure if this issue you are mentioning is related the Kibana that is not loading at all from one node, or that the Kibana Monitoring page is not loading correctly but Kibana itself loads properly.
Because the timeout is relative to an API for the monitoring but other API calls are returning 200 so seems to runs correctly.

If it's a Slack Monitoring issue, where you can't load the monitoring page, let me know and I can ping monitoring team directly

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.