Welcome Everybody,
I’m trying to setup secure ES cluster with Kibana. I was following the steps outlined in the Elasticsearch security: Best practices to keep your data safe webinar
Everything seems to be fine except the final step in kibana setup.
I’m not getting the expected login screen but rather the following error:
{"statusCode":401,"error":"Unauthorized","message":"[security_exception] missing authentication token for REST request [/.kibana/doc/config%3A6.8.0], with { header={ WWW-Authenticate={ 0=\"Bearer realm=\\\"security\\\"\" & 1=\"ApiKey\" & 2=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}
I can curl elastic securely with elastic user:
curl -k https://elasticsearch:9200/_cat/nodes?v -uelastic
I have also checked xpack access with kibana user
curl -k -u kibana https://elasticsearch:9200/_xpack?pretty
All looks fine, still GUI refuses to start.
ES cluster and kibana are all running in containers. My docker run for kibana looks as follows.
docker run -d --name=kibana \
-p 5601:5601/tcp \
-e ELASTICSEARCH_SSL_VERIFICATIONMODE=none \
-e ELASTICSEARCH_URL=https://elasticsearch \
-e ELASTICSEARCH_USERNAME=kibana \
-e ELASTICSEARCH_PASSWORD=changeme \
kibana/kibana-oss:6.8.0
Docker logs:
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["warning","elasticsearch","config","deprecation"],"pid":1,"message":"Config key "url" is deprecated. It has been replaced with "hosts""}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:kibana@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:elasticsearch@6.8.0","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:console@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:interpreter@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:metrics@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:tile_map@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:timelion@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:elasticsearch@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0:5601"}
Then each request to kibana result in the below entry:
{"type":"response","@timestamp":"2019-11-22T13:04:29Z","tags":[],"pid":1,"method":"get","statusCode":401,"req":{"url":"/app/kibana","method":"get","headers":{"host":"kibana:5601","connection":"keep-alive","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.8.130.88","userAgent":"10.8.130.88"},"res":{"statusCode":401,"responseTime":26,"contentLength":9},"message":"GET /app/kibana 401 26ms - 9.0B"}
Additional info:
$ curl -k -u kibana:changeme https://elasticsearch:9200/_xpack/license
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 338 100 338 0 0 5545 0 --:--:-- --:--:-- --:--:-- 5728
{
"license" : {
"status" : "active",
"uid" : "80d7e6ce-3489-4973-bc4c-0413ebb22d74",
"type" : "basic",
"issue_date" : "2019-10-10T12:14:04.080Z",
"issue_date_in_millis" : 1570709644080,
"max_nodes" : 1000,
"issued_to" : "es-ct-security",
"issuer" : "elasticsearch",
"start_date_in_millis" : -1
}
}
$ curl -k -u kibana:changeme https://elasticsearch:9200/_xpack/security/_authenticate?pretty
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 330 100 330 0 0 7878 0 --:--:-- --:--:-- --:--:-- 8250
{
"username" : "kibana",
"roles" : [
"kibana_system"
],
"full_name" : null,
"email" : null,
"metadata" : {
"_reserved" : true
},
"enabled" : true,
"authentication_realm" : {
"name" : "reserved",
"type" : "reserved"
},
"lookup_realm" : {
"name" : "reserved",
"type" : "reserved"
}
}
Kibana was working fine before enabling security.
What can be the issue?
How can I troubleshoot this further?
PS. I also have nginx in front of Kibana to enable 443 for it but even with nginx down, the issue remains.
UI loads fine now and I can log into ES.