User in xpack security not added

I am trying to add user in kibana after running the ./bin/kibana-keystore add elasticsearch.username --allow-root and password command in Kibana folder. But when I restart kibana after setting username and password, i get this error. What is the issue?

 log   [19:05:34.591] [warning][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml
  log   [19:05:34.608] [error][status][plugin:reporting@7.4.0] Status changed from uninitialized to red - [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }
  log   [19:05:34.632] [error][status][plugin:security@7.4.0] Status changed from green to red - [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }
  log   [19:05:34.709] [warning][reporting] Reporting plugin self-check failed. Please check the Kibana Reporting settings. [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } } :: {"path":"/_cluster/settings","query":{"include_defaults":true},"statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}","wwwAuthenticateDirective":"Basic realm=\"security\" charset=\"UTF-8\""}
  log   [19:06:04.351] [warning][license][xpack] License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } } :: {"path":"/_xpack","statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}","wwwAuthenticateDirective":"Basic realm=\"security\" charset=\"UTF-8\""}

This is my kibana.yml

server.port: 5601
server.host: "x.x.x.x"
elasticsearch.hosts: ["http://localhost:9200"]
elasticsearch.username: "*****"
elasticsearch.password: "****"
xpack.security.encryptionKey: "esq_Mehak_Virtual_Machine_security_setup"

Hi @Mehak_Bhargava,

Can you enable debug logging by setting:

logging.verbose: true

in your kibana.yml?

After this, restart Kibana, and provide that log output.

Are you able to connect to Elasticsearch using the username and password you configured if you hit it manually? What does this return?

curl -u kibana:YOUR_PASSWORD_HERE http://localhost:9200

(replace "YOUR_PASSWORD_HERE" with the same password you provided the ./bin/kibana-keystore command.)

  log   [19:39:12.838] [warning][license][xpack] License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } } :: {"path":"/_xpack","statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}","wwwAuthenticateDirective":"Basic realm=\"security\" charset=\"UTF-8\""}
  log   [19:39:13.190] [debug][browser-driver][reporting] Browser installed at /home/mehak/Documents/kibana-7.4.0-linux-x86_64/data/headless_shell-linux/headless_shell
  log   [19:39:13.191] [debug][reporting] Browser type: chromium
  log   [19:39:13.191] [debug][reporting] Chromium sandbox disabled: false
  log   [19:39:13.223] [warning][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml
  log   [19:39:13.232] [error][status][plugin:reporting@7.4.0] Status changed from uninitialized to red - [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }
  log   [19:39:13.235] [debug][esqueue][queue-worker][reporting] k7wavqvn09pb8f5fe5dam00b - Created worker for reporting jobs
  log   [19:39:13.247] [debug][reporting] Running on os "linux", distribution "Ubuntu Linux", release "18.04"
  log   [19:39:13.249] [error][status][plugin:security@7.4.0] Status changed from green to red - [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }
  log   [19:39:13.333] [warning][reporting] Reporting plugin self-check failed. Please check the Kibana Reporting settings. [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } } :: {"path":"/_cluster/settings","query":{"include_defaults":true},"statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}","wwwAuthenticateDirective":"Basic realm=\"security\" charset=\"UTF-8\""}
  log   [19:39:15.260] [debug][plugin] Checking Elasticsearch version
  log   [19:39:17.835] [debug][plugin] Checking Elasticsearch version
  log   [19:39:20.437] [debug][plugin] Checking Elasticsearch version
  log   [19:39:23.007] [debug][plugin] Checking Elasticsearch version
  log   [19:39:25.581] [debug][plugin] Checking Elasticsearch version
  log   [19:39:28.153] [debug][plugin] Checking Elasticsearch version
  log   [19:39:30.719] [debug][plugin] Checking Elasticsearch version
  log   [19:39:33.300] [debug][plugin] Checking Elasticsearch version
  log   [19:39:35.868] [debug][plugin] Checking Elasticsearch version
  log   [19:39:38.465] [debug][plugin] Checking Elasticsearch version
  log   [19:39:41.059] [debug][plugin] Checking Elasticsearch version
  log   [19:39:42.841] [debug][license][xpack] Calling [data] Elasticsearch _xpack API. Polling frequency: 30001
  log   [19:39:42.923] [warning][license][xpack] License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } } :: {"path":"/_xpack","statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}","wwwAuthenticateDirective":"Basic realm=\"security\" charset=\"UTF-8\""}

Yes, i am able to connect to ES

mehak@mehak-VirtualBox:~/Documents/kibana-7.4.0-linux-x86_64$ curl -u kibana:***** http://localhost:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [kibana]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"failed to authenticate user [kibana]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

@Larry_Gregory my password for elastic is different than the password I set for kibana. Do they both have to be same? if so, how do I change kibana password now?

When i did elasticsearch-password interactive, I used one password for all elasticsearch, logstash, apmsystem,beatsystem,kibana. So do I enter this password while setting the kibana username password too?

Also, this is my log from ES. I cannot open kibana GUi and delete indexes from there. Is there a curl command through which I can delete the index without going to kibana's UI? Because the issue might be that it doesnt have space as thresholdMonitor suggests in log

[2020-03-17T13:24:29,194][INFO ][o.e.c.r.a.DiskThresholdMonitor] [mehak-VirtualBox] low disk watermark [93%] exceeded on [uszTm_0tR26zJa0KI9beBw][mehak-VirtualBox][/home/mehak/Documents/elasticsearch-7.4.0/data/nodes/0] free: 809.2mb[5.8%], replicas will not be assigned to this node
[2020-03-17T13:24:29,624][INFO ][o.e.x.s.a.AuthenticationService] [mehak-VirtualBox] Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana]
[2020-03-17T13:24:32,203][INFO ][o.e.x.s.a.AuthenticationService] [mehak-VirtualBox] Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana]
[2020-03-17T13:24:34,802][INFO ][o.e.x.s.a.AuthenticationService] [mehak-VirtualBox] Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana]
[2020-03-17T13:24:37,392][INFO ][o.e.x.s.a.AuthenticationService] [mehak-VirtualBox] Authentication of [kibana] was terminated by realm [reserved] 

Nope, they can (and should) be set to different values.

If disk space isn't the issue, then this is suggesting that your password isn't correct.

The API for deleting indices is here: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-delete-index.html

ex (deletes an index called "twitter"):

curl -X DELETE "localhost:9200/twitter"

I tried the curl command again and entered the correct password. Should I reset password to try this process again? I changed my password in kibana.yml, and then ran the curl command you mentioned below and still I had same error.

I tried the curl command to delete index and got this error

mehak@mehak-VirtualBox:~/Documents/kibana-7.4.0-linux-x86_64$ curl -X DELETE "localhost:9200/ob-webapi"
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/ob-webapi]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/ob-webapi]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

@Larry_Gregory what should I try to figure out the issue?

password for this command for ex-
./bin/elasticsearch-setup-passwords interactive - "abc"
./bin/kibana-keystore add elasticsearch.username --allow-root - "def"
./bin/kibana-keystore add elasticsearch.password --allow-root - "def"

Above is my passwords example pattern.

Below is my elasticsearch.yml

xpack.monitoring.enabled: false

#cluster.name: my-application
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 93%
cluster.routing.allocation.disk.watermark.high: 95%

xpack.security.enabled: true
discovery.type: single-node

You didn't provide credentials, which is why you got this error. Sorry my example didn't make that explicit:

missing authentication credentials for REST request

Can you authenticate to elasticsearch using your elastic superuser account? If you know that password, then you can use that to reset the kibana user account:

curl -X POST "localhost:9200/_security/user/kibana/_password?pretty" -H 'Content-Type: application/json' -u elastic:YOUR_ELASTIC_USER_PASSWORD -d'
{
  "password" : "YOUR_NEW_PASSWORD"
}
'

This is the Change Password API, documented here: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-change-password.html

I was able to delete index by providing credentials.
But after entering this curl command in terminal, the command woulnt exit

mehak@mehak-VirtualBox:~/Documents/kibana-7.4.0-linux-x86_64$ curl -X POST "localhost:9200/_security/user/kibana/_password?pretty" -H 'Content-Type:application/json' -u elastic:**** -d'
> {
>  "password" : "****"
> }
> 

And when i posted this on kibana dev tools UI like this-

POST "localhost:9200/_security/user/kibana/_password?pretty" -H 'Content-Type: application/json' -u elastic:**** -d'
{
  "password" : "****"
}

I got this error

{
  "error": "no handler found for uri [/%22localhost:9200/_security/user/kibana/_password?pretty%22%20-H%20%27Content-Type:%20application/json%27%20-u%20elastic:elastic%20-d%27&pretty] and method [POST]"
}

So can you access Kibana now, if you have access to Kibana dev tools?

Dev Tools does not expect you to provide the URL to elasticsearch. You can instead do something like this when logged into Kibana as the elastic user:

POST /_security/user/kibana/_password?pretty
{
  "password" : "****"
}

Yes, i can access Kibana now. But I tried the grok debugger tool right now with sample data and grok pattern, And It didnt even show what kind of error it is as it usually does on the down right hand side bottom in red. The same data and grok i ran in grok debugger website and it worked perfectly. So It is not entirely working.

I commented out the xpack security line in elasticsearch.yml and the lines I added in kibana.yml to remove these security features as I no longer require them. How can i get my system back to before xpack as I want my kibana and logstash working.

kibana.yml

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
logging.verbose: true

ES.yml

xpack.monitoring.enabled: false
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 93%
cluster.routing.allocation.disk.watermark.high: 95%

Do I have to run commands to remove the elastic's superuser and kibana username password?

@Larry_Gregory, Hi I am not able to run kibana as before xpack. This is my logs from Kibana. Please tell how to revert back my settings as Elastic and Kibana arent running as expected.

  log   [19:33:19.782] [debug][license][xpack] Calling [data] Elasticsearch _xpack API. Polling frequency: 30001
  log   [19:33:20.633] [debug][plugin] Checking Elasticsearch version
  log   [19:33:20.742] [debug][kibana-monitoring][monitoring] Received Kibana Ops event data
  log   [19:33:20.742] [debug][kibana-monitoring][monitoring] Received Kibana Ops event data
  log   [19:33:21.328] [debug][kibana-monitoring][monitoring] Received Kibana Ops event data
  log   [19:33:21.330] [debug][kibana-monitoring][monitoring] Received Kibana Ops event data
  ops   [19:33:22.247]  memory: 225.4MB uptime: 0:40:47 load: [0.04 0.15 0.14] delay: 0.179
  log   [19:33:23.137] [debug][plugin] Checking Elasticsearch version
  log   [19:33:25.643] [debug][plugin] Checking Elasticsearch version
  log   [19:33:25.742] [debug][kibana-monitoring][monitoring] Received Kibana Ops event data
  log   [19:33:25.742] [debug][kibana-monitoring][monitoring] Received Kibana Ops event data
  log   [19:33:26.333] [debug][kibana-monitoring][monitoring] Received Kibana Ops event data
  log   [19:33:26.335] [debug][kibana-monitoring][monitoring] Received Kibana Ops event data
  ops   [19:33:27.250]  memory: 225.9MB uptime: 0:40:52 load: [0.12 0.17 0.14] delay: 0.361

Used to be like below while Kibana was running

log   [01:50:58.281] [debug][plugins] Initializing plugin code@7.4.0
  log   [01:50:58.282] [info][status][plugin:code@7.4.0] Status changed from uninitialized to green - Ready
  log   [01:50:58.283] [debug][plugins] Initializing plugin data@kibana
  log   [01:50:58.283] [info][status][plugin:data@7.4.0] Status changed from uninitialized to green - Ready
  log   [01:50:58.284] [debug][plugins] Initializing plugin visualizations@kibana
  log   [01:50:58.284] [info][status][plugin:visualizations@7.4.0] Status changed from uninitialized to green - Ready
  log   [01:50:58.285] [debug][plugins] Initializing plugin interpreter@kibana
  log   [01:50:58.287] [info][status][plugin:interpreter@7.4.0] Status changed from uninitialized to green - Ready
  log   [01:50:58.287] [debug][plugins] Initializing plugin tile_map@kibana
  log   [01:50:58.288] [info][status][plugin:tile_map@7.4.0] Status changed from uninitialized to green - Ready
  log   [01:50:58.288] [debug][plugins] Initializing plugin task_manager@7.4.0
  log   [01:50:58.290] [info][status][plugin:task_manager@7.4.0] Status changed from uninitialized to green - Ready
  log   [01:50:58.290] [debug][plugins] Initializing plugin maps@7.4.0
  log   [01:50:58.293] [info][status][plugin:maps@7.4.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:50:58.295] [debug][plugins] Initializing plugin canvas@7.4.0
  log   [01:50:58.304] [info][status][plugin:canvas@7.4.0] Status changed from uninitialized to green - Ready
  log   [01:50:58.305] [debug][plugins] Initializing plugin license_management@7.4.0
  log   [01:50:58.310] [info][status][plugin:license_management@7.4.0] Status changed from uninitialized to green - Ready
  log   [01:50:58.310] [debug][plugins] Initializing plugin cloud@7.4.0
  log   [01:50:58.312] [info][status][plugin:cloud@7.4.0] Status changed from uninitialized to green - Ready
  log   [01:50:58.312] [debug][plugins] Initializing plugin index_management@7.4.0
  log   [01:50:58.313] [info][status][plugin:index_management@7.4.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:50:58.320] [debug][plugins] Initializing plugin console@kibana
  log   [01:50:58.323] [info][status][plugin:console@7.4.0] Status changed from uninitialized to green - Ready
  log   [01:50:58.324] [debug][plugins] Initializing plugin console_extensions@7.4.0
  log   [01:50:58.325] [info][status][plugin:console_extensions@7.4.0] Status changed from uninitialized to green - Ready
  log   [01:50:58.326] [debug][plugins] Initializing plugin index_lifecycle_management@7.4.0
  log   [01:50:58.327] [info][status][plugin:index_lifecycle_management@7.4.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:50:58.336] [debug][plugins] Initializing plugin kuery_autocomplete@7.4.0
  log   [01:50:58.338] [debug][plugins] Initializing plugin metrics@kibana
  log   [01:50:58.340] [info][status][plugin:metrics@7.4.0] Status changed from uninitialized to green - Ready
  log   [01:50:58.341] [debug][plugins] Initializing plugin infra@7.4.0
  log   [01:50:58.361] [info][status][plugin:infra@7.4.0] Status changed from uninitialized to green - Ready
  log   [01:50:58.362] [debug][plugins] Initializing plugin rollup@7.4.0
  log   [01:50:58.363] [info][status][plugin:rollup@7.4.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:50:58.367] [debug][plugins] Initializing plugin siem@7.4.0
  log   [01:50:58.369] [info][siem] Plugin initializing
  log   [01:50:58.388] [info][siem] Plugin done initializing
  log   [01:50:58.388] [info][status][plugin:siem@7.4.0] Status changed from uninitialized to green - Ready
  log   [01:50:58.389] [debug][plugins] Initializing plugin remote_clusters@7.4.0
  log   [01:50:58.392] [info][status][plugin:remote_clusters@7.4.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:50:58.395] [debug][plugins] Initializing plugin cross_cluster_replication@7.4.0
  log   [01:50:58.397] [info][status][plugin:cross_cluster_replication@7.4.0] Status changed from uninitialized to yellow

In my pipeline logstash, I first had two indexes defined - 'dispatcher-logs' and 'rbac-logs'. Kibana showed two indexes made, but the second index was renamed to %{fields} shown below.

green  .security-7              open WLeXvDv-QK6pyLSZ4sQurw 1 0    6  0  19.6kb 2020-03-17T18:49:59.932Z
yellow mehak                    open N6ttCpigRuG6RRMH3kpD3g 1 1    0  0    283b 2019-10-30T20:09:08.123Z
green  .kibana_task_manager_1   open ZgPpIuUyRzanjFOiom3FSg 1 0    2  0  12.6kb 2020-01-23T21:39:49.525Z
green  .apm-agent-configuration open WAdlGHwxSZCF3EQ8CzkeIA 1 0    0  0    283b 2019-11-14T23:07:14.386Z
yellow dispatcher-logs          open 0ohSnv6pQvaILmvLKpgWpg 1 1 1114  0 209.4kb 2020-03-23T19:50:48.372Z
green  .kibana_1                open F22MPzabTfyEamcqzkN_SQ 1 0   72 10 132.7kb 2020-01-23T22:21:02.901Z
yellow %{[fields][tags]}        open K1a62fHLT7-ztuc7OTsG0g 1 1   12  0    14kb 2020-03-23T19:50:49.340Z
yellow dispatcher-app           open -k6LF3X3R6y8UmjDQf64BQ 1 1   68  0  68.9kb 2020-03-17T01:24:44.791Z
green  .tasks                   open 8HzlaiA-QZmYPGBhotuSJA 1 0    1  0   6.3kb 2019-11-18T19:52:09.491Z
yellow ngta-common              open fn79VV5CRje9B8Pp9Loosg 1 1   11  0  61.7kb 2020-03-17T01:50:16.699Z

@Larry_Gregory Kibana has stopped making indexes when I run logstash and filebeat. In Logstash debug, I see the JSON but its not available in kibana. How do I fix my kibana?