Kibana failing with requestTimeout error

Hi,

I am using kibana-oss 7.0.1 in kubernetes environment.
Elasticsearch pods are taking around 2 minutes to come up and form the cluster.

Till then, kibana pod fails with the below error as it is not able to reach Elasticsearch:

{"type":"log","@timestamp":"2019-07-05T12:05:42Z","tags":["status","plugin:timelion@undefined","info"],"pid":1,"state":"green",message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-07-05T12:05:45Z","tags":["status","plugin:elasticsearch@undefined","error"],"pid":1,"state":"rd","message":"Status changed from yellow to red - Request Timeout after 3000ms","prevState":"yellow","prevMsg":"Waiting for Elaticsearch"}
{"type":"log","@timestamp":"2019-07-05T12:06:31Z","tags":["status","plugin:elasticsearch@undefined","info"],"pid":1,"state":"gren","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-07-05T12:07:01Z","tags":["fatal","root"],"pid":1,"message":"{ Error: Request Timeout after 3000ms\n at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15\n at Timeout. (/usr/share/kiana/node_modules/elasticsearch/src/lib/transport.js:384:7)\n at ontimeout (timers.js:436:11)\n at tryOnTimeout (timers.js300:5)\n at listOnTimeout (timers.js:263:5)\n at Timer.processTimers (timers.js:223:10)\n status: undefined,\n displayNme: 'RequestTimeout',\n message: 'Request Timeout after 30000ms',\n body: undefined }"}

FATAL Error: Request Timeout after 30000ms

After the pod restarts once, second time it came up properly as by this time ES is up and is reachable.

I tried increasing the timeout to 3 min using elasticsearch.requestTimeout: 180000 so that the first time itself, pod doesnt restart due to timeout error.
With this setting, again kibana failed with a different FATAL error:

,"message":"{ [master_not_discovered_exception] null :: {"path":"/.kibana","query":{},"statusCode":503,"response":"{\"error\":{\"root_cause\":[{\"type\":\"master_not_discovered_exception\",\"reason\":null}],\"type\":\"master_not_discovered_exception\",\"reason\":null},\"status\":503}"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)\n status: 503,\n displayName: 'ServiceUnavailable',\n message: '[master_not_discovered_exception] null',\n path: '/.kibana',\n query: {},\n body:\n { error:\n { root_cause: [Array],\n type: 'master_not_discovered_exception',\n reason: null },\n status: 503 },\n statusCode: 503,\n response:\n '{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}',\n toString: [Function],\n toJSON: [Function] }"}

This happened much before 3 min and by this time, cluster hadn't got formed.

I tried setting elasticsearch.startupTimeout: 180000 as well. I ended up with the same error. Can you tell me what parameter should be added to delay kibana contacting ES? so that I can avoid this issue?
Hoping for a quick reply.

It looks like Elasticsearch is sending the 503 code to Kibana because of its master_not_discovered_exception. Kibana should handle this better, but you need to address the Elasticsearch cluster problems.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.