Hi,
I am using kibana-oss 7.0.1 in kubernetes environment.
Elasticsearch pods are taking around 2 minutes to come up and form the cluster.
Till then, kibana pod fails with the below error as it is not able to reach Elasticsearch:
{"type":"log","@timestamp":"2019-07-05T12:05:42Z","tags":["status","plugin:timelion@undefined","info"],"pid":1,"state":"green",message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-07-05T12:05:45Z","tags":["status","plugin:elasticsearch@undefined","error"],"pid":1,"state":"rd","message":"Status changed from yellow to red - Request Timeout after 3000ms","prevState":"yellow","prevMsg":"Waiting for Elaticsearch"}
{"type":"log","@timestamp":"2019-07-05T12:06:31Z","tags":["status","plugin:elasticsearch@undefined","info"],"pid":1,"state":"gren","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-07-05T12:07:01Z","tags":["fatal","root"],"pid":1,"message":"{ Error: Request Timeout after 3000ms\n at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15\n at Timeout. (/usr/share/kiana/node_modules/elasticsearch/src/lib/transport.js:384:7)\n at ontimeout (timers.js:436:11)\n at tryOnTimeout (timers.js300:5)\n at listOnTimeout (timers.js:263:5)\n at Timer.processTimers (timers.js:223:10)\n status: undefined,\n displayNme: 'RequestTimeout',\n message: 'Request Timeout after 30000ms',\n body: undefined }"}FATAL Error: Request Timeout after 30000ms
After the pod restarts once, second time it came up properly as by this time ES is up and is reachable.
I tried increasing the timeout to 3 min using elasticsearch.requestTimeout: 180000 so that the first time itself, pod doesnt restart due to timeout error.
With this setting, again kibana failed with a different FATAL error:
,"message":"{ [master_not_discovered_exception] null :: {"path":"/.kibana","query":{},"statusCode":503,"response":"{\"error\":{\"root_cause\":[{\"type\":\"master_not_discovered_exception\",\"reason\":null}],\"type\":\"master_not_discovered_exception\",\"reason\":null},\"status\":503}"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)\n status: 503,\n displayName: 'ServiceUnavailable',\n message: '[master_not_discovered_exception] null',\n path: '/.kibana',\n query: {},\n body:\n { error:\n { root_cause: [Array],\n type: 'master_not_discovered_exception',\n reason: null },\n status: 503 },\n statusCode: 503,\n response:\n '{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}',\n toString: [Function],\n toJSON: [Function] }"}
This happened much before 3 min and by this time, cluster hadn't got formed.
I tried setting elasticsearch.startupTimeout: 180000 as well. I ended up with the same error. Can you tell me what parameter should be added to delay kibana contacting ES? so that I can avoid this issue?
Hoping for a quick reply.