Hi people,
Got an annoying problem here with Kibana 6.8.0, setting up a test cluster with HTTPS and Xpack enabled. The Elastic cluster is running ok and our application can use this just fine.
But Kibana can only run 1 minute then it crashes with
{"type":"log","@timestamp":"2019-12-12T12:33:06Z","tags":["error","task_manager"],"pid":2851,"message":"Failed to poll for work: [security_exception] missing authentication token for REST request [/.kibana_task_manager/_doc/_search?ignore_unavailable=true], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } } :: {"path":"/.kibana_task_manager/_doc/_search","query":{"ignore_unavailable":true},"body":"{\"query\":{\"bool\":{\"must\":[{\"term\":{\"type\":\"task\"}},{\"bool\":{\"must\":[{\"terms\":{\"task.taskType\":[\"maps_telemetry\",\"vis_telemetry\"]}},{\"range\":{\"task.attempts\":{\"lte\":3}}},{\"range\":{\"task.runAt\":{\"lte\":\"now\"}}},{\"range\":{\"kibana.apiVersion\":{\"lte\":1}}}]}}]}},\"size\":10,\"sort\":{\"task.runAt\":{\"order\":\"asc\"}},\"seq_no_primary_term\":true}","statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/.kibana_task_manager/_doc/_search?ignore_unavailable=true]\",\"header\":{\"WWW-Authenticate\":[\"Bearer realm=\\\"security\\\"\",\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/.kibana_task_manager/_doc/_search?ignore_unavailable=true]\",\"header\":{\"WWW-Authenticate\":[\"Bearer realm=\\\"security\\\"\",\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}},\"status\":401}","wwwAuthenticateDirective":"Bearer realm=\"security\", ApiKey, Basic realm=\"security\" charset=\"UTF-8\""}"}
If I restart Kibana, it works again for 1 minute. When it is working I can use all the functions just fine, monitoring, devtools and so on.
My Kibana.yml looks like this
server.port: 5601
server.host: "testkibana.domain.com"
server.name: "testkibana.domain.com"
elasticsearch.hosts: ["https://estestdata-0.domain.com:9200"]
elasticsearch.sniffInterval: 60000
elasticsearch.sniffOnConnectionFault: true
elasticsearch.sniffOnStart: false
elasticsearch.preserveHost: true
kibana.index: ".kibana"
elasticsearch.username: "kibana"
elasticsearch.password: "password"
xpack.monitoring.elasticsearch.username: "elastic"
xpack.monitoring.elasticsearch.password: "password"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/domain_com.crt
server.ssl.key: /etc/kibana/Private.key
elasticsearch.ssl.certificate: /etc/kibana/domain_com.crt
elasticsearch.ssl.key: /etc/kibana/Private.key
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/CA.crt" ]
elasticsearch.ssl.verificationMode: none
elasticsearch.requestHeadersWhitelist: [ authorization ]
logging.dest: /etc/kibana/kibana.log
logging.quiet: true
We use a real certificate for our servers, so it is not a generated by Elasticsearch one. Have another issue with the Elasticsearch - that is why SSL verification is "none"
Hope anyone has a hit to what to fix/try
Best regards
Thomas