Where is the Kibana password set?

oskrdt · January 19, 2021, 4:53pm

I was transferred the ownership of a server with elasticsearch and kibana running, where I was not involved on the setup process.
I was given a username and password to access kibana, which seems to have being manually setup, that works fine to access kibana GUI. It seems that the password is only required on kibana, since I can interact with elasticsearch without any authentication.
The problem came when I was trying to setup the security for both elasticsearch and kibana, since it seems the elastic user password does work on elasticsearch, but it's not working on kibana. Furthermore, it seems that no matter that I commented out the xpack.security.enabled: true on the elasticsearch.yml and even added xpack.security.enabled: false to try it out, but kibana GUI is still requesting authentication... so this makes me wonder, where the kibana password is set? where is that stored?

wylie · January 20, 2021, 6:36pm

It sounds like you may need to refresh yourself on the Kibana + Elasticsearch security docs. I can try to point you in the right direction.

All user accounts are stored in Elasticsearch. We require certain roles to be able to log in to Kibana: maybe your elastic user has been downgraded from a superuser?

There are built-in user accounts when you install the stack, and you are meant to configure the passwords at install time. In particular, the elastic user is a superuser, and depending on your version you may also have a kibana user or kibana_system user.

You are supposed to update kibana.yml to configure security, and this is how Kibana connects to Elasticsearch.

You should also check what roles your users have.

oskrdt · January 21, 2021, 5:49am

Thanks, that clarifies the highlevel doubt I had, but it raised some others.

How is it possible that elasticsearch has the security turned off (connections are not requiring authentication) but kibana is requiring auth even when I disable it on the yml file.
How is it possible that after creating the built-in user accounts following the procedure, I can't use the elastic super user to log into kibana.

It seems that kibana decided to have his own authentication method independent from elasticsearch users and settings.

wylie · January 21, 2021, 4:36pm

I think you should look at the user roles on the elastic user. GET /_security/user/elastic will show you if it is still a superuser.

Secondly, Kibana will only show an authentication screen if the ES connection requires it. This might be the obvious question, but are you sure your Kibana instance is connected to the right ES? Or maybe you updated the elasticsearch.yml file without restarting ES?

oskrdt · January 21, 2021, 10:10pm

I did some checks to confirm that I'm connecting to the correct ES instance and I'm restarting them after changing the yml file.
I enabled security in order to perform the query you told me and it seems the elastic user has indeed superuser role. Actually, here is the output of the users query

{
    "elastic": {
        "username": "elastic",
        "roles": [
            "superuser"
        ],
        "full_name": null,
        "email": null,
        "metadata": {
            "_reserved": true
        },
        "enabled": true
    },
    "kibana": {
        "username": "kibana",
        "roles": [
            "kibana_system"
        ],
        "full_name": null,
        "email": null,
        "metadata": {
            "_deprecated": true,
            "_deprecated_reason": "Please use the [kibana_system] user instead.",
            "_reserved": true
        },
        "enabled": true
    },
    "kibana_system": {
        "username": "kibana_system",
        "roles": [
            "kibana_system"
        ],
        "full_name": null,
        "email": null,
        "metadata": {
            "_reserved": true
        },
        "enabled": true
    },
    "logstash_system": {
        "username": "logstash_system",
        "roles": [
            "logstash_system"
        ],
        "full_name": null,
        "email": null,
        "metadata": {
            "_reserved": true
        },
        "enabled": true
    },
    "beats_system": {
        "username": "beats_system",
        "roles": [
            "beats_system"
        ],
        "full_name": null,
        "email": null,
        "metadata": {
            "_reserved": true
        },
        "enabled": true
    },
    "apm_system": {
        "username": "apm_system",
        "roles": [
            "apm_system"
        ],
        "full_name": null,
        "email": null,
        "metadata": {
            "_reserved": true
        },
        "enabled": true
    },
    "remote_monitoring_user": {
        "username": "remote_monitoring_user",
        "roles": [
            "remote_monitoring_collector",
            "remote_monitoring_agent"
        ],
        "full_name": null,
        "email": null,
        "metadata": {
            "_reserved": true
        },
        "enabled": true
    }
}

With the security enabled I did the following tests:

Tried to open Kibana using elastic credentials: the result is that I get the authentication box again as if I was using a wrong user or password. No messages on the Kibana server side.
Tried to open Kibana using kibanaadmin credentials provided by the former owner: the result is that I pass the authentication box, but I get the following error message
{"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [kibanaadmin] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }"}

Where is the kibanaadmin credentials setup?

wylie · January 21, 2021, 11:00pm

Weird issue, but we've seen issues like this caused by unusual kibana.yml settings. Please provide a snippet of your kibana.yml settings, with sensitive info removed. Also, are you running with a proxy or other networking layer between you and Kibana?
That's a custom user, not a built-in user. see docs

oskrdt · January 21, 2021, 11:11pm

Here is the content of my kibana.yml file

# Kibana is served by a back end server. This setting specifies the port to use.
#server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
server.maxPayloadBytes: 104857600

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
#elasticsearch.hosts: ["http://localhost:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana_system"
elasticsearch.password: "*************"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid

# Enables you to specify a file where Kibana stores log output.
#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

# Increase the max file zise to import
savedObjects.maxImportPayloadBytes: 104857600

xpack.security.enabled: true

oskrdt · January 21, 2021, 11:18pm

Another info regarding my Elasticsearch/Kibana instances that may be relevant:

I moved from 7.9 to 7.10 version
I moved from OSS to basic

wylie · January 21, 2021, 11:41pm

Did you see my question about proxy or any other network layer between you and Kibana?

oskrdt · January 22, 2021, 12:02am

I see it now!
It is using a nginx reverse proxy and digging on the config I see the /etc/nginx/htpasswd.users with the definition of kibanaadmin user.
Now I understand why this wasn't a usual setup of the stack, and why I was so lost with the documentation.
Thanks for the support that pointed to the root cause!

system · February 19, 2021, 12:02am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.