Kibana server is not yet ready after adding basic licensed x-pack security

[edk]

I'm trying to setup minimal security for Elasticsearch and kibana on an existing test instance running 6.8.23 on a single server running both Elasticsearch and kibana, deployed on windows.

Enable Elasticsearch security features | Elasticsearch Guide [6.8] | Elastic

I've gotten Elasticsearch to start following the above instructions plus my Elasticsearch logs showed it failed to start until I also added xpack.security.transport.ssl.enabled: true to Elasticsearch.yml

I auto created the built-in user passwords: bin/Elasticsearch-setup-passwords auto
And confirmed that without the elastic user and password called out, the curl get fails:
curl -XGET "http://localhost:9200/_cluster/health?pretty"
but succeeds with them:
curl -XGET -u elastic:password "http://localhost:9200/_cluster/health?pretty"

Adding the kibana username and password from the buit-in user setup to the kibana.yml:
Elasticsearch.username: "kibana"
Elasticsearch.password: "password"
AND in the keystore:
bin/kibana-keystore create
bin/kibana-keystore add Elasticsearch.username
bin/kibana-keystore add Elasticsearch.password

Restarting Kibana, the service seems to start but the kibana page http://localhost:5601 is stuck on the "Kibana is not yet ready" page.

I've tried using "kibana_system" as the username, and trying the elastic username and password which some other deployment instructions mentioned.
I've also tried to set some additional x-pack configurations in the kibana.yml:
xpack.security.encryptionKey: "something_at_least_32_characters"
xpack.security.enabled: true
That I found in these documentations:

Configuring Security in Kibana | Kibana Guide [6.8] | Elastic

Security Settings in Kibana | Kibana Guide [6.8] | Elastic

If I revert all of the configs, kibana opens again as it did previously without the security login for intial setup with the elastic username and password.

I feel like I'm very close based on the other forum topics similar to this issue, but those issues all seem to be resolved by setting the built-in passwords and configuring kibana.
My kibana.yml doesn't have specific configuration calling out any logging but I'm guessing it's erroring and I just don't know where to find the logs. Or I've missed a very simple step.

My hope was to get the minimal setup configured and so I can begin looking into the TSL/SSL configuration for a Production cluster.

[warkolm (Mark Walkom)]

Welcome to our community!

I just wanted to make a quick note that 6.8 is EOL and no longer supported. You should really upgrade as a matter of urgency.

@edk

Restarting Kibana, the service seems to start but the kibana page http://localhost:5601 is stuck on the "Kibana is not yet ready" page.

It'd be useful if you could share your Kibana and Elasticsearch logs.

Please also remember to format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you

[edk]

I got the Kibana logs outputting to a file and there's definitely a problem, that I'm not understanding from the log:

{"type":"log","@timestamp":"2022-03-02T19:00:11Z","tags":["status","plugin:spaces@6.8.23","error"],"pid":47488,"state":"red","message":"Status changed from yellow to red - action [indices:admin/get] is unauthorized for user [kibana]: [security_exception] action [indices:admin/get] is unauthorized for user [kibana]","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2022-03-02T19:00:11Z","tags":["fatal","root"],"pid":47488,"message":"{ [security_exception] action [indices:admin/get] is unauthorized for user [kibana] :: {\"path\":\"/.uatnocmonitoring\",\"query\":{\"include_type_name\":true},\"statusCode\":403,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"action [indices:admin/get] is unauthorized for user [kibana]\\\"}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"action [indices:admin/get] is unauthorized for user [kibana]\\\"},\\\"status\\\":403}\"}\n    at respond (D:\\ELK\\kibana-6.8.23\\node_modules\\elasticsearch\\src\\lib\\transport.js:308:15)\n    at checkRespForFailure (D:\\ELK\\kibana-6.8.23\\node_modules\\elasticsearch\\src\\lib\\transport.js:267:7)\n    at HttpConnector.<anonymous> (D:\\ELK\\kibana-6.8.23\\node_modules\\elasticsearch\\src\\lib\\connectors\\http.js:166:7)\n    at IncomingMessage.wrapper (D:\\ELK\\kibana-6.8.23\\node_modules\\elasticsearch\\node_modules\\lodash\\lodash.js:4991:19)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)\n  status: 403,\n  displayName: 'AuthorizationException',\n  message:\n   'action [indices:admin/get] is unauthorized for user [kibana]: [security_exception] action [indices:admin/get] is unauthorized for user [kibana]',\n  path: '/.uatnocmonitoring',\n  query: { include_type_name: true },\n  body:\n   { error:\n      { root_cause: [Array],\n        type: 'security_exception',\n        reason:\n         'action [indices:admin/get] is unauthorized for user [kibana]' },\n     status: 403 },\n  statusCode: 403,\n  response:\n   '{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:admin/get] is unauthorized for user [kibana]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:admin/get] is unauthorized for user [kibana]\"},\"status\":403}',\n  toString: [Function],\n  toJSON: [Function],\n  isBoom: true,\n  isServer: false,\n  data: null,\n  output:\n   { statusCode: 403,\n     payload:\n      { message:\n         'action [indices:admin/get] is unauthorized for user [kibana]: [security_exception] action [indices:admin/get] is unauthorized for user [kibana]',\n        statusCode: 403,\n        error: 'Forbidden' },\n     headers: {} },\n  [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/forbidden' }"}

Where .uatnocmonitoring is the kibana index:

kibana.index: ".uatnocmonitoring"

I also wanted to follow-up that I have confirmed that the password for the built-in kibana user is correct and can launch:

curl -XGET -u kibana:my_password "http://localhost:9200/_cluster/health?pretty"

That's in addition to confirm the elastic built-in user and password is also correct.

Most discussions I've found around this issue seem to be when then the password is actually incorrect. That doesn't seem to be the issue in my case since the API calls are working

I will say that I do have plans to upgrade to 7.x, however the hope was that I could fairly easily get the x-pack security in place first.

[warkolm (Mark Walkom)]

@edk

Where .uatnocmonitoring is the kibana index:

I would suggest changing that to the default unless you have a good reason for using it.

Otherwise, the default role for the kibana user doesn't have access to the index, and you'd need to update that.

[edk]

Changing the kibana.index back to

kibana.index: ".kibana"

was the solution. I'm not sure why it was custom, but I removed the security from elastic

xpack.security.enabled: false

exported the saved objects, re-enabled security, set the kibana index back to .kibana, signed into kibana with the kibana username and password successfully. Once in I could complete the users and roles setup and all I had to do was import my saved object so the .kibana index had all the index patterns, searches, visualizations and dashboards from the previous kibana index.