Kibana error after using key ca certs instead of plaintext password

Hello Team,

I have recently implemented certs and keys for Kibana (ELK 6.8.0) instead of username and password, but getting, but getting missing authentication token. Details logs are as:

  log   [10:50:19.005] [info][status][plugin:cross_cluster_replication@6.8.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [10:50:19.015] [info][status][plugin:translations@6.8.0] Status changed from uninitialized to green - Ready
  log   [10:50:19.025] [info][status][plugin:upgrade_assistant@6.8.0] Status changed from uninitialized to green - Ready
  log   [10:50:19.036] [info][status][plugin:uptime@6.8.0] Status changed from uninitialized to green - Ready
  log   [10:50:19.038] [info][status][plugin:oss_telemetry@6.8.0] Status changed from uninitialized to green - Ready
  log   [10:50:19.048] [info][status][plugin:metrics@6.8.0] Status changed from uninitialized to green - Ready
  log   [10:50:19.177] [info][status][plugin:timelion@6.8.0] Status changed from uninitialized to green - Ready
  log   [10:50:20.060] [error][status][plugin:xpack_main@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.061] [error][status][plugin:graph@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.061] [error][status][plugin:spaces@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.062] [error][status][plugin:security@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.062] [error][status][plugin:searchprofiler@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.062] [error][status][plugin:ml@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.063] [error][status][plugin:tilemap@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.063] [error][status][plugin:watcher@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.064] [error][status][plugin:grokdebugger@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.064] [error][status][plugin:logstash@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.064] [error][status][plugin:beats_management@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.065] [error][status][plugin:maps@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.065] [error][status][plugin:index_management@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.065] [error][status][plugin:index_lifecycle_management@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.066] [error][status][plugin:rollup@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.066] [error][status][plugin:remote_clusters@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.066] [error][status][plugin:cross_cluster_replication@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.067] [error][status][plugin:elasticsearch@6.8.0] Status changed from yellow to red - Authentication Exception
  log   [10:50:20.087] [warning][license][xpack] License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] missing authentication token for REST request [/_xpack], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } } :: {"path":"/_xpack","statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/_xpack]\",\"header\":{\"WWW-Authenticate\":[\"Bearer realm=\\\"security\\\"\",\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/_xpack]\",\"header\":{\"WWW-Authenticate\":[\"Bearer realm=\\\"security\\\"\",\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}},\"status\":401}","wwwAuthenticateDirective":"Bearer realm=\"security\", ApiKey, Basic realm=\"security\" charset=\"UTF-8\""}
  log   [10:50:20.088] [error][status][plugin:xpack_main@6.8.0] Status changed from red to red - [security_exception] missing authentication token for REST request [/_xpack], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }
  log   [10:50:20.089] [error][status][plugin:graph@6.8.0] Status changed from red to red - [security_exception] missing authentication token for REST request [/_xpack], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }

Kibana.yml look like:

server.name: "Kibana(PAD)"
elasticsearch.url: 'https://x.x.x.x:9200/'

#elasticsearch.username: "kibana"
#elasticsearch.password: "supersecert_password"

xpack.security.enabled: true
elasticsearch.ssl.certificate: config/certs/client.cer
elasticsearch.ssl.key: config/certs/client.key
elasticsearch.ssl.certificateAuthorities: config/certs/client-ca.cer
elasticsearch.ssl.verificationMode: certificate
xpack.reporting.encryptionKey: "something_secret"

Please help me out on this, or do we need license for this(i know its very silly question)

Thanks in Advance!!!
Akshay.

In order to use client TLS authentication to authenticate the kibana user to Elasticsearch with a client certificate, you need to configure a PKI realm in Elasticsearch and generate a client certificate for the kibana user .

PKI realms are not available in the basic license ( see the security section in the basic column here ) so you;d need a Gold license or above.

Yes, i have enabled PKI realm in elasticsearch, and genrated certificates:

xpack.security.authc.realms.pki1.type pki

Can you suggest who we can generate certificate for specially kibana user.
Just for info i have followed this link to step up so.

You can follow the instructions on that blog to achieve what you want, I believe it's detailed enough. If you've fully followed all the relevant instructions there and it still doesn't work for you, please share your elasticsearch.yml and kibana.yml

Here are config files for Kibana:

server.host: '0.0.0.0'
server.name: "Kibana(PAD)"
elasticsearch.url: 'https://elasticsearch_host:9200/'

elasticsearch.username: "kibana"
elasticsearch.password: "supersecret_password"
xpack.security.enabled: true
elasticsearch.ssl.certificate: config/certs/client.cer
elasticsearch.ssl.key: config/certs/client.key
elasticsearch.ssl.certificateAuthorities: config/certs/client-ca.cer
elasticsearch.ssl.verificationMode: certificate
xpack.reporting.encryptionKey: "something_secret"

Elasticsearch.yml:

xpack.security.transport.ssl.verification_mode:certificate
xpack.security.transport.ssl.truststore.path:certs/elastic-certificates.p12
xpack.security.transport.ssl.keystore.path:certs/elastic-certificates.p12
xpack.security.transport.ssl.enabled:true
xpack.security.http.ssl.truststore.path:certs/elastic-certificates.p12
xpack.security.http.ssl.keystore.path:certs/elastic-certificates.p12
xpack.security.http.ssl.enabled:true
xpack.security.http.ssl.client_authentication:optional
xpack.security.enabled:true
xpack.security.authc.run_as.enabled:true
xpack.security.authc.realms.pki1.type:pki
xpack.monitoring.history.duration:7d
xpack.monitoring.collection.enabled:true
node.name:${HOSTNAME}
node.master:true
node.data:false
indices.memory.index_buffer_size:40%
http.cors.enabled:true
http.cors.allow-origin:*
ES_JAVA_OPTS:-Xms512m -Xmx512m
discovery.zen.ping.unicast.hosts:es-master
discovery.zen.minimum_master_nodes:2
DISABLED_xpack.monitoring.exporters.id1.type:http
DISABLED_xpack.monitoring.exporters.id1.host:http://es-monitoring.es.svc.cluster.local:9200
cluster.routing.allocation.awareness.attributes:rack_id
cluster.name:rancher-es

Is the error from kibana still

  log   [10:50:20.087] [warning][license][xpack] License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] missing authentication token for REST request [/_xpack], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } } :: {"path":"/_xpack","statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/_xpack]\",\"header\":{\"WWW-Authenticate\":[\"Bearer realm=\\\"security\\\"\",\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/_xpack]\",\"header\":{\"WWW-Authenticate\":[\"Bearer realm=\\\"security\\\"\",\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}},\"status\":401}","wwwAuthenticateDirective":"Bearer realm=\"security\", ApiKey, Basic realm=\"security\" charset=\"UTF-8\""}
  log   [10:50:20.088] [error][status][plugin:xpack_main@6.8.0] Status changed from red to red - [security_exception] missing authentication token for REST request [/_xpack], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }
  log   [10:50:20.089] [error][status][plugin:graph@6.8.0] Status changed from red to red - [security_exception] missing authentication token for REST request [/_xpack], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }

or has it changed now ? Any logs from the Elasticsearch side ?

Btw the elasticsearch.yml you have shared is not valid yaml, there is no space between the keys and the values, your Elasticsearch instance wouldn't even start like this.

There are no error from elasticsearch side and still error are same.

There is no problem for space, because I'm running that on Rancher environment.