Unable to login to Kibana

Elastic Stack Elasticsearch
1

Trying to setup ES with LDAP. Getting the follow error while logging in Kibana.

{"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [*****] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"}

2

There is not enough info in your message for us to help you. What is your configuration ? What error are there in the logs ?

3

Sorry for the late reply. Please find the below details.

I tried using AD and the same error persists.

/var/elasticsearch/mycluster.log

[2020-01-20T11:19:54,127][DEBUG][o.e.x.s.a.l.ActiveDirectorySessionFactory] [node-3] Resolved 143 LDAP groups

*******Lists all the LDAP groups it found **************************

[2020-01-20T11:19:54,128][DEBUG][o.e.x.s.a.l.ActiveDirectorySessionFactory] [node-3] Resolved 0 meta-data fields [{}] for user [CN=admuser,OU=Some OU1,OU=Some OU2,OU=Some OU3OU=SomeOU4,DC=SomeDC,DC=com]
[2020-01-20T11:20:18,476][DEBUG][o.e.x.s.a.l.LdapRealm ] [node-3] realm [myad] authenticated user [admuser], with roles [[BB_Server_Admins]]

/etc/elasticsearch/elasticsearch.yml

xpack:
security:
authc:
realms:
active_directory:
myad:
order: 0
domain_name: domain.com
url: ldap://SOMEServer.Domain.COM:389
bind_dn: "CN=some\, user, OU=SomeOU,OU=SomeOU2,OU=SomeOU3,OU=SomeOU4,OU=SomeOU5,DC=SomeDC,DC=com"
follow_referrals: false

Role Mapping.

PUT /_security/role_mapping/BB_Server_Admins
{
"roles" : [ "BB_Server_Admins" ],
"rules" : { "field" : {
"groups" : "CN=BB Server Admins,OU=Groups,OU=SomeOU,OU=SomeOU,OU1=SomeOU2,OU=SomeOU3,DC=SomeDC,DC=com"
} },
"enabled": true
}

Roles (created in GUI)

{
"BB_Server_Admins" : {
"cluster" : [
"all"
],
"indices" : [
{
"names" : [
"wazuh-monitoring-3.x-",
"apm-",
"wazuh-alerts-3.x-"
],
"privileges" : [
"all"
],
"allow_restricted_indices" : false
}
],
"applications" : [
{
"application" : "kibana-.kibana",
"privileges" : [
"all"
],
"resources" : [
""
]
}
],
"run_as" : ,
"metadata" : { },
"transient_metadata" : {
"enabled" : true
}
}
}

ERROR while logging in kibana

{"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [admuser] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"}

4

Your log file indicates that authentication worked, but you're showing an error from Kibana that shows that it failed.

Can you explain the mismatch there? What triggered that successful authentication in the logs, and where is the log from the Kibana failure?

5

Hello Tim,

Please find below Kibana logs.

Kibana logs where going to stdout by default. So changed it to /var/log/kibana/somfile.log in /etc/kibana/kibana.yml

{"type":"response","@timestamp":"2020-01-20T23:47:15Z","tags":[],"pid":30484,"method":"post","statusCode":204,"r eq":{"url":"/api/security/v1/login","method":"post","headers":{"host":"10.0.0.0:5601","connection":"keep-aliv e","content-length":"62","accept":"application/json, text/plain, /","kbn-version":"7.3.0","user-agent":"Mozill a/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"," content-type":"application/json;charset=UTF-8","origin":"https://10.0.0.0:5601","sec-fetch-site":"same-origin ","sec-fetch-mode":"cors","referer":"https://10.0.0.0:5601/login","accept-encoding":"gzip, deflate, br","acce pt-language":"en-US,en;q=0.9"},"remoteAddress":"10.1.1.1","userAgent":"10.1.1.1","referer":"https://10.0.0.0:5601/login"},"res":{"statusCode":204,"responseTime":18,"contentLength":9},"message":"POST /api/security/v1 /login 204 18ms - 9.0B"}

10.1.1.1 is DHCP server addrs

{"type":"error","@timestamp":"2020-01-20T23:47:16Z","tags":["connection","client","error"],"pid":30484,"level":" error","error":{"message":"140541772941184:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate u nknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 46\n","name":"Error","stack":"Err or: 140541772941184:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/ openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 46\n"},"message":"140541772941184:error:14094416:SSL rou tines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 46\n"}

{"type":"log","@timestamp":"2020-01-20T23:47:16Z","tags":["info","authentication"],"pid":30484,"message":"Authen tication attempt failed: [security_exception] unable to authenticate user [admuser] for REST request [/_se curity/_authenticate], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2 ="Basic realm=\"security\" charset=\"UTF-8\"" } } }"}

Bearer realm=\"security\\This is not my realm

{"type":"response","@timestamp":"2020-01-20T23:47:16Z","tags":[],"pid":30484,"method":"get","statusCode":401,"re q":{"url":"/","method":"get","headers":{"host":"10.0.0.0:5601","connection":"keep-alive","upgrade-insecure-re quests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrom e/79.0.3945.130 Safari/537.36","sec-fetch-user":"?1","accept":"text/html,application/xhtml+xml,application/xml;q =0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"same-origin","sec -fetch-mode":"navigate","referer":"https://10.0.0.1:5601/login","accept-encoding":"gzip, deflate, br","accept -language":"en-US,en;q=0.9"},"remoteAddress":"10.1.1.1","userAgent":"10.1.1.1","referer":"https://10.0.0.0:5601/login"},"res":{"statusCode":401,"responseTime":27,"contentLength":9},"message":"GET / 401 27ms - 9.0B"}

{"type":"response","@timestamp":"2020-01-20T23:47:16Z","tags":[],"pid":30484,"method":"get","statusCode":302,"re q":{"url":"/favicon.ico","method":"get","headers":{"host":"10.0.0.0:5601","connection":"keep-alive","user-age nt":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safar i/537.36","accept":"image/webp,image/apng,image/,/*;q=0.8","sec-fetch-site":"same-origin","sec-fetch-mode":"no -cors","referer":"https://10.0.0.0:5601/","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q =0.9"},"remoteAddress":"10.1.1.1","userAgent":"10.1.1.1","referer":"https://10.0.0.0:5601/"},"res":{"st atusCode":302,"responseTime":3,"contentLength":9},"message":"GET /favicon.ico 302 3ms - 9.0B"}

{"type":"response","@timestamp":"2020-01-20T23:47:16Z","tags":[],"pid":30484,"method":"get","statusCode":200,"re q":{"url":"/login?next=%2Ffavicon.ico","method":"get","headers":{"host":"10.0.0.0:5601","connection":"keep-al ive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0. 3945.130 Safari/537.36","accept":"image/webp,image/apng,image/,/*;q=0.8","sec-fetch-site":"same-origin","sec-f etch-mode":"no-cors","referer":"https://10.0.0.0:5601/","accept-encoding":"gzip, deflate, br","accept-languag e":"en-US,en;q=0.9"},"remoteAddress":"10.1.1.1","userAgent":"10.1.1.1","referer":"https://10.0.0.0:5601 /"},"res":{"statusCode":200,"responseTime":374,"contentLength":9},"message":"GET /login?next=%2Ffavicon.ico 200 374ms - 9.0B"}

6

Sorry, that wasn't what I meant.
You showed us two things:

  • An Elasticsearch log that clearly showed authentication working
  • A Kibana error message that showed authentication not working

You didn't explain how those 2 things were related.

You've now given us this Kibana log:

{"type":"log","@timestamp":"2020-01-20T23:47:16Z",
 "tags":["info","authentication"],"pid":30484,
 "message":"Authentication attempt failed: [security_exception] unable to authenticate user [admuser] for REST request [/_se curity/_authenticate], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2 ="Basic realm=\"security\" charset=\"UTF-8\""
} } }"}

Can you show us the Elasticsearch log for the same timestamp?

7

Hello Tim,

Below are the Kibana and ES logs, when I press the login button in Kibana console.

Note: there is a time difference of 1hr between ES and Kibana logs. I tried to change the timezone in Kibana console but its not reflecting in logs.

Kibana Log

{"type":"log","@timestamp":"2020-01-22T04:55:23Z","tags":["info","authentication"],"pid":13793,"message":"Authentication attempt fail
ed: [security_exception] unable to authenticate user [admuser] for REST request [/_security/_authenticate], with { header={ WWW
-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"}

ES log

[2020-01-22T15:55:23,006][DEBUG][o.e.x.s.a.l.LdapRealm ] [node-3] realm [myrlm] authenticated user [admuser], with roles [[BB_Server_Admins]]

This is what I get when I use elastic user

{"type":"log","@timestamp":"2020-01-22T05:05:06Z","tags":["info","audit","security","saved_objects_authorization_success"],"pid":13793,"username":"elastic","action":"get","types":["config"],"args":{"type":"config","id":"7.3.0","options":{}},"eventType":"saved_objects_authorization_success","message":"elastic authorized to get config"}

My Configuration files are as below

Summary

===================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please consult the documentation for further information on configuration options:

https://www.elastic.co/guide/en/elasticsearch/reference/index.html

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: my-cluster

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: node-3

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

Path to log files:

path.logs: /var/log/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 10.0.0.3

Set a custom port for HTTP:

#http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when this node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

discovery.seed_hosts: ["10.0.0.1", "10.0.0.2", "10.0.0.3"]

Bootstrap the cluster using an initial set of master-eligible nodes:

cluster.initial_master_nodes:

  • 10.0.0.1
  • 10.0.0.2
  • 10.0.0.3

For more information, consult the discovery and cluster formation module documentation.

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

#gateway.recover_after_nodes: 3

For more information, consult the gateway module documentation.

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

#action.destructive_requires_name: true
xpack.security.enabled: true

Transport layer

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/elasticsearch-3.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/elasticsearch-3.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]

HTTP layer

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.key: /etc/elasticsearch/certs/elasticsearch-3.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/elasticsearch-3.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]

#AD

xpack:
security:
authc:
realms:
active_directory:
myrlm:
order: 0
domain_name: domain.com
url: ldap://server.domain.COM:389
bind_dn: "CN=Some Account\, BB user,OU=someOU1,OU=someOU2,OU=someOU3,OU=someOU4,OU=someOU5,DC=someDC,DC=com"
follow_referrals: false

Summary

Kibana is served by a back end server. This setting specifies the port to use.

#server.port: 5601

Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.

The default is 'localhost', which usually means remote machines will not be able to connect.

To allow connections from remote users, set this parameter to a non-loopback address.

server.host: "0.0.0.0"

Enables you to specify a path to mount Kibana at if you are running behind a proxy.

Use the server.rewriteBasePath setting to tell Kibana if it should remove the basePath

from requests it receives, and to prevent a deprecation warning at startup.

This setting cannot end in a slash.

#server.basePath: ""

Specifies whether Kibana should rewrite requests that are prefixed with

server.basePath or require that they are rewritten by your reverse proxy.

This setting was effectively always false before Kibana 6.3 and will

default to true starting in Kibana 7.0.

#server.rewriteBasePath: false

The maximum payload size in bytes for incoming server requests.

#server.maxPayloadBytes: 1048576

The Kibana server's name. This is used for display purposes.

server.name: "myname"

The URLs of the Elasticsearch instances to use for all your queries.

elasticsearch.hosts: ["https://10.0.0.1:9200", "https://10.0.0.2:9200", "https://10.0.0.3:9200"]

When this setting's value is true Kibana uses the hostname specified in the server.host

setting. When the value of this setting is false, Kibana uses the hostname of the host

that connects to this Kibana instance.

#elasticsearch.preserveHost: true

Kibana uses an index in Elasticsearch to store saved searches, visualizations and

dashboards. Kibana creates a new index if the index doesn't already exist.

#kibana.index: ".kibana"

The default application to load.

#kibana.defaultAppId: "home"

If your Elasticsearch is protected with basic authentication, these settings provide

the username and password that the Kibana server uses to perform maintenance on the Kibana

index at startup. Your Kibana users still need to authenticate with Elasticsearch, which

is proxied through the Kibana server.

#elasticsearch.username: "kibana"
#elasticsearch.password: "pass"

Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.

These settings enable SSL for outgoing requests from the Kibana server to the browser.

#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

Optional settings that provide the paths to the PEM-format SSL certificate and key files.

These files validate that your Elasticsearch backend uses the same key files.

#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

Optional setting that enables you to specify a path to the PEM file for the certificate

authority for your Elasticsearch instance.

#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

To disregard the validity of SSL certificates, change this setting's value to 'none'.

#elasticsearch.ssl.verificationMode: full

Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of

the elasticsearch.requestTimeout setting.

#elasticsearch.pingTimeout: 1500

Time in milliseconds to wait for responses from the back end or Elasticsearch. This value

must be a positive integer.

elasticsearch.requestTimeout: 30000

List of Kibana client-side headers to send to Elasticsearch. To send no client-side

headers, set this value to (an empty list).

#elasticsearch.requestHeadersWhitelist: [ authorization ]

Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten

by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.

#elasticsearch.customHeaders: {}

Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.

#elasticsearch.shardTimeout: 30000

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.

#elasticsearch.startupTimeout: 5000

Logs queries sent to Elasticsearch. Requires logging.verbose set to true.

#elasticsearch.logQueries: false

Specifies the path where Kibana creates the process ID file.

#pid.file: /var/run/kibana.pid

Enables you specify a file where Kibana stores log output.

logging.dest: /var/log/kibana/some.log

Set the value of this setting to true to suppress all logging output.

#logging.silent: false

Set the value of this setting to true to suppress all logging output other than error messages.

#logging.quiet: false

Set the value of this setting to true to log all events, including system usage information

and all requests.

#logging.verbose: false

Set the interval in milliseconds to sample system and process performance

metrics. Minimum is 100ms. Defaults to 5000.

#ops.interval: 5000

Specifies locale to be used for all localizable strings, dates and number formats.

Supported languages are the following: English - en , by default , Chinese - zh-CN .

#i18n.locale: "en"
xpack.security.audit.enabled: true
xpack.security.enabled: true
xpack.security.encryptionKey: "*****************"

Elasticsearch from/to Kibana

elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/ca.crt"]
elasticsearch.ssl.certificate: "/etc/kibana/certs/elasticsearch-3.crt"
elasticsearch.ssl.key: "/etc/kibana/certs/elasticsearch-3.key"

Browser from/to Kibana

server.ssl.enabled: true
server.ssl.certificate: "/etc/kibana/certs/elasticsearch-3.crt"
server.ssl.key: "/etc/kibana/certs/elasticsearch-3.key"
elasticsearch.username: "elastic"
elasticsearch.password: "************"

8

Sorry, it was my mistake.

We had 3 master nodes and I had done the AD configuration only in one node hoping that it will automatically replicate to other nodes.
Configuration file was made the same in all the node and bind user password was provided in all - Kibana is now authenticating properly with AD.

Thank you

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.