Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]

Elastic Stack Elasticsearch
1

Hello everyone,

I have my two node cluster (8.3.2.1) - working. I generated the certificates of each element (elastic-vm-1, elastic-vm2, kibana, logstash) including DNS and IP for each certificate.

Kibana - that when I change the user it is put in disabled mode - it seems that it has some authentication problem when connecting to elasticsearch.

kibana logs:
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-12T10:57:40.699+02:00","message":"Authentication attempt failed: {\" error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [kibana_system] for REST request [/_security/_authenticate]\ ",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\" Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [ kibana_system] for REST request [/_security/_authenticate]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\" UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","log": {"level":"INFO","logger":"plugins.security.authentication"},"process":{"pid":705},"trace":{"id":"9a227874dd779f220b70a5f835779620"},"transaction ":{"id":"79bcebc151899fc5"}}

Log Elastic:

[2022-07-12T11:12:01,414][INFO ][o.e.x.s.a.RealmsAuthenticator] [elastic-vm-1] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]

Kibana settings:

server.port: 5601
server.host: "10.172.211.14"
server.publicBaseUrl: "https://10.172.211.30:5601"
server.name: "kibana"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/kibana.crt
server.ssl.key: /etc/kibana/kibana.key
elasticsearch.hosts: ["https://10.172.211.30:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "passworddddd"
elasticsearch.ssl.certificate: /etc/kibana/kibana.crt
elasticsearch.ssl.key: /etc/kibana/kibana.key
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/ca.crt" ]
pid.file: /run/kibana/kibana.pid
xpack.security.encryptionKey: "shkhKFBqncoQJCPQJwoixioynoqwneioAOjdnoihfafsfda"

I have created a profile with no configuration limits and assigned a user with that profile to the kibaba configuration. Same mistake.

It seems that it is not from elasticserach permissions. I have read that it could be because of the certificates - but I generated all of them again including the IP in each of them and I managed to solve another problem... but now I have this one.

If in elasticsearch.hosts I put in DNS of the host - then kibana doesn't start... and if I put the IP this connection fails.

2

Is it possible that I need to include the CA key with some such command?

/usr/share/kibana/bin/kibana-keystore add xpack.security.transport.ssl.keystore.secure_password
/usr/share/kibana/bin/kibana-keystore add xpack.security.transport.ssl.truststore.secure_password

I tried it and then if it does not raise the kibana service

3

Verify if kibana_system user can authenticate directly to Elasticsearch. For example:

curl -s -X GET --cacert "/etc/elasticsearch/https-ca.crt" -H "Authorization: Basic `echo -n kibana_system:passworddddd|base64 -`" https://10.172.211.30:9200/_security/_authenticate?pretty=true

Also, I notice you have two authentication methods configured in kibana.yml. Is that expected, or did you mean to comment one of them out?

Reserved username/password authentication:

elasticsearch.username: "kibana_system"
elasticsearch.password: "passworddddd"

TLS client authentication:

elasticsearch.ssl.certificate: /etc/kibana/kibana.crt
elasticsearch.ssl.key: /etc/kibana/kibana.key
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/ca.crt" ]

If using username/password, verify you are using the correct password via the curl example. You are required to use the kibana_system password that was auto-generated by Elasticsearch at startup, or you can manually reset it with bin/elasticsearch-reset-password -i -u kibana_system.

If using TLS client authentication, it requires the HTTPS listener in Elasticsearch to trust /etc/kibana/ca.crt, and a PKI realm with the same /etc/kibana/ca.crt, and a role mapping rule to map the PKI realm and/or Kibana client cert Subject DN's Common Name to the kibana_system role.

4

Hi Justin

Muchas gracias por tu ayuda.

The username and password seem correct.

root@kibana:~# curl -s -X GET --cacert "/etc/kibana/ca.crt" -H "Authorization: Basic `echo -n kibana_system:***********|base64 -`" https://10.172.211.30:9200/_security/_authenticate?pretty=true
{
  "username" : "kibana_system",
  "roles" : [
    "kibana_system"
  ],
  "full_name" : null,
  "email" : null,
  "metadata" : {
    "_reserved" : true
  },
  "enabled" : true,
  "authentication_realm" : {
    "name" : "reserved",
    "type" : "reserved"
  },
  "lookup_realm" : {
    "name" : "reserved",
    "type" : "reserved"
  },
  "authentication_type" : "realm"
}

Ok, thanks for the clarification. I've removed TLS for now.
But leaving only user and pass does not start the service.

Browser:

Kibana server is not ready yet.

Log:

{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-12T20:26:26.644+02:00","message":"Unable to retrieve version information from Elasticsearch nodes. unable to verify the first certificate","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":143222},"trace":{"id":"883a7844b7bae1d67546bfcda5bc8b30"},"transaction":{"id":"a69aa01f63bf81ba"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-12T20:26:27.159+02:00","message":"Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell","log":{"level":"INFO","logger":"plugins.screenshotting.chromium"},"process":{"pid":143222},"trace":{"id":"883a7844b7bae1d67546bfcda5bc8b30"},"transaction":{"id":"a69aa01f63bf81ba"}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-07-12T20:26:27.160+02:00","message":"Enabling the Chromium sandbox provides an additional layer of protection.","log":{"level":"WARN","logger":"plugins.screenshotting.chromium"},"process":{"pid":143222},"trace":{"id":"883a7844b7bae1d67546bfcda5bc8b30"},"transaction":{"id":"a69aa01f63bf81ba"}}

This problem that if the service is alive but the system considers that it is not reliable... I solved it by changing the certificates including also its DNS and its IP and altering the instruction - until it works.

IP
elasticsearch.hosts: ["https://10.172.211.30:9200"]
hostname
elasticsearch.hosts: ["https://elastic-vm-1:9200"]
DNS
elasticsearch.hosts: ["https://elastic-vm-1.prso.net:9200"]

But only with with just the username and password - it doesn't work either way.

In parallel I would like to understand the TLS authentication system - it seems more robust. Can you tell me how it is orchestrated?

How do I tell elasticsearch that this kibana.crt certificate corresponds to which role?

Thank you so much again Justin

5

In general, these options for Kibana have worked for me.

  • username/password for kibana_system user
  • service token for kibana_system user
  • TLS client key+cert for kibana_system user

The first two require Kibana to trust the CA that issued the HTTPS server CA for Elasticsearch.

The third requires that, plus Elasticsearch HTTPS listener must trust the CA that issued the Kibana HTTPS client certificate. HTTPS server and HTTPS client certs have different contents for Subject DN and Subject Alt Name. Servers have hostnames and IPs, clients have usernames.

If you are trying to use the Kibana cert with dnsName and ipAddress, but as a TLS client cert for login to Elasticsearch, it sounds like you are trying to use it for both purposes, as a server and a client.

In general, there could be minimum of 3 CAs for signing certs:

  • HTTPS server certs, with dnsName and ipAddress
  • HTTPS client certs, with Subject DN containing CN=username
  • Transport/TLS client+server certs, with optional Subject DN containing CN=clustering+nodeid, for Elasticsearch clustering

It is possible to use a single CA, but best practice is for clients to be from non-public CAs. You may even want different CAs to separate internal users (ex:kibana_system) vs external users (ex: superuser1). For those user CAs, you probably also want them in a PKI domain each, and use role mapping from CN=username and/or PKI realm name to roles. For a CN=kibana_system client cert, role map that username or it's PKI realm name to kibana_system role.

6

finaly it's working fine with user/pass (ca.crt)

root@kibana:~# cat /etc/kibana/kibana.yml
server.port: 5601
server.host: 0.0.0.0
server.publicBaseUrl: "https://kibana:5601"
server.name: "kibana"
elasticsearch.hosts: ["https://elastic-vm-1:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "passwordddd"

elasticsearch.ssl.certificate: /etc/kibana/kibana.crt
elasticsearch.ssl.key: /etc/kibana/kibana.key
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/ca.crt" ]
*elasticsearch.ssl.verificationMode: none*

xpack.encryptedSavedObjects.encryptionKey: af8b566a4cb71901d454513d957f78b1
xpack.reporting.encryptionKey: 66737e3b09f0ccf8e45452be7f18c37e
xpack.security.encryptionKey: 81442d45453879c3b8afdbd9d2c79f4b

Thanks

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.