Kibana UI - missing authentication token for REST

Welcome Everybody,

I’m trying to setup secure ES cluster with Kibana. I was following the steps outlined in the Elasticsearch security: Best practices to keep your data safe webinar

Everything seems to be fine except the final step in kibana setup.

I’m not getting the expected login screen but rather the following error:
{"statusCode":401,"error":"Unauthorized","message":"[security_exception] missing authentication token for REST request [/.kibana/doc/config%3A6.8.0], with { header={ WWW-Authenticate={ 0=\"Bearer realm=\\\"security\\\"\" & 1=\"ApiKey\" & 2=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}

I can curl elastic securely with elastic user:
curl -k https://elasticsearch:9200/_cat/nodes?v -uelastic

I have also checked xpack access with kibana user
curl -k -u kibana https://elasticsearch:9200/_xpack?pretty

All looks fine, still GUI refuses to start.

ES cluster and kibana are all running in containers. My docker run for kibana looks as follows.

docker run -d --name=kibana \
-p 5601:5601/tcp \
-e ELASTICSEARCH_SSL_VERIFICATIONMODE=none \
-e ELASTICSEARCH_URL=https://elasticsearch \
-e ELASTICSEARCH_USERNAME=kibana \
-e ELASTICSEARCH_PASSWORD=changeme \
kibana/kibana-oss:6.8.0

Docker logs:
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["warning","elasticsearch","config","deprecation"],"pid":1,"message":"Config key "url" is deprecated. It has been replaced with "hosts""}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:kibana@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:elasticsearch@6.8.0","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:console@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:interpreter@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:metrics@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:tile_map@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:timelion@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["status","plugin:elasticsearch@6.8.0","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-11-22T13:03:35Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0:5601"}

Then each request to kibana result in the below entry:
{"type":"response","@timestamp":"2019-11-22T13:04:29Z","tags":[],"pid":1,"method":"get","statusCode":401,"req":{"url":"/app/kibana","method":"get","headers":{"host":"kibana:5601","connection":"keep-alive","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.8.130.88","userAgent":"10.8.130.88"},"res":{"statusCode":401,"responseTime":26,"contentLength":9},"message":"GET /app/kibana 401 26ms - 9.0B"}

Additional info:

$ curl -k -u kibana:changeme https://elasticsearch:9200/_xpack/license
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   338  100   338    0     0   5545      0 --:--:-- --:--:-- --:--:--  5728
{
  "license" : {
    "status" : "active",
    "uid" : "80d7e6ce-3489-4973-bc4c-0413ebb22d74",
    "type" : "basic",
    "issue_date" : "2019-10-10T12:14:04.080Z",
    "issue_date_in_millis" : 1570709644080,
    "max_nodes" : 1000,
    "issued_to" : "es-ct-security",
    "issuer" : "elasticsearch",
    "start_date_in_millis" : -1
  }
}

$ curl -k -u kibana:changeme https://elasticsearch:9200/_xpack/security/_authenticate?pretty
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   330  100   330    0     0   7878      0 --:--:-- --:--:-- --:--:--  8250
{
  "username" : "kibana",
  "roles" : [
    "kibana_system"
  ],
  "full_name" : null,
  "email" : null,
  "metadata" : {
    "_reserved" : true
  },
  "enabled" : true,
  "authentication_realm" : {
    "name" : "reserved",
    "type" : "reserved"
  },
  "lookup_realm" : {
    "name" : "reserved",
    "type" : "reserved"
  }
}

Kibana was working fine before enabling security.

What can be the issue?
How can I troubleshoot this further?

PS. I also have nginx in front of Kibana to enable 443 for it but even with nginx down, the issue remains.

Hey @m0rt, when you're using the default version of Elasticsearch with security enabled, you'll also want to be using the default version of Kibana. It looks like you're using the kibana/kibana-oss:6.8.0 docker image, can you switch to the default distribution and ensure you're using the official docker image per https://www.elastic.co/guide/en/kibana/6.8/docker.html?

docker run -d --name=kibana \
-p 5601:5601/tcp \
-e ELASTICSEARCH_SSL_VERIFICATIONMODE=none \
-e ELASTICSEARCH_URL=https://elasticsearch \
-e ELASTICSEARCH_USERNAME=kibana \
-e ELASTICSEARCH_PASSWORD=changeme \
docker.elastic.co/kibana/kibana:6.8.0

Thanks a lot Brandon! So simple. Looks much better now :slight_smile: UI loads fine now and I can log into ES.

Thanks!

1 Like