I just upgraded ELK from 7.1.0 to 7.5.0 and Kibana fails to start with
{"type":"log","@timestamp":"2020-01-22T17:27:54Z","tags":["error","elasticsearch","data"],"pid":23107,"message":"Request error, retrying\nGET http://localhost:9200/_xpack => socket hang up"}
{"type":"log","@timestamp":"2020-01-22T17:27:55Z","tags":["info","plugins-system"],"pid":23107,"message":"Starting [8] plugins: [security,licensing,code,timelion,features,spaces,translations,data]"}
{"type":"log","@timestamp":"2020-01-22T17:27:55Z","tags":["warning","plugins","licensing"],"pid":23107,"message":"License information could not be obtained from Elasticsearch for the [data] cluster. [security_exception] missing authentication credentials for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication credentials for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication credentials for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}"}
when having the following two options enabled:
elasticsearch.sniffOnStart: true
elasticsearch.sniffOnConnectionFault: true
Any idea what I am doing wrong ?
The complete Kibana config follows:
server.port: 5601
server.host: 0.0.0.0
server.name: kibana
kibana.index: ".kibana"
kibana.defaultAppId: "discover"
elasticsearch.hosts: ["http://node1.test.com:9200", "http://node2.test.com:9200", "http://node3.test.com:9200", "http://node4.test.com:9200", "http://node5.test.com:9200"]
elasticsearch.pingTimeout: 1500
elasticsearch.requestTimeout: 30000
elasticsearch.logQueries: true
elasticsearch.sniffOnStart: true
elasticsearch.sniffOnConnectionFault: true
elasticsearch.username: "kibana"
elasticsearch.password: "XXX"
logging.dest: /var/log/kibana.log
logging.verbose: false
xpack.security.enabled: true
xpack.monitoring.enabled: true
xpack.monitoring.ui.enabled: true
xpack.security.encryptionKey: "XXX"
If I remove elasticsearch.sniffOnStart: true then all works fine.