Kibana not starting for enabling AD security

Elastic Stack Kibana
1

Dear All,
Recently enabled elasticsearch with AD with xpack and able to authenticate successfully via curl command. Kibana is not properly starting up, any help will be appreciated. My intention is to have Kibana secured with AD

{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","root"],"pid":10664,"message":"setting up root"}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","server"],"pid":10664,"message":"setting up server"}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","http"],"pid":10664,"message":"starting NotReady server"}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","http","server","Kibana"],"pid":10664,"message":"registering route handler for [/core]"}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","elasticsearch-service"],"pid":10664,"message":"Setting up elasticsearch service"}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","elasticsearch-service"],"pid":10664,"message":"Creating elasticsearch clients"}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","plugins-service"],"pid":10664,"message":"Setting up plugins service"}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","plugins-discovery"],"pid":10664,"message":"Discovering plugins..."}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","plugins-discovery"],"pid":10664,"message":"Scanning \"/usr/share/kibana/src/plugins\" for plugin sub-directories..."}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","plugins-discovery"],"pid":10664,"message":"Scanning \"/usr/share/kibana/x-pack/plugins\" for plugin sub-directories..."}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","plugins-discovery"],"pid":10664,"message":"Scanning \"/usr/share/kibana/plugins\" for plugin sub-directories..."}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","plugins-discovery"],"pid":10664,"message":"Scanning \"/usr/share/kibana-extra\" for plugin sub-directories..."}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","plugins-discovery"],"pid":10664,"message":"Successfully discovered plugin \"translations\" at \"/usr/share/kibana/x-pack/plugins/translations\""}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","plugins","translations"],"pid":10664,"message":"\"/usr/share/kibana/x-pack/plugins/translations/server\" does not export \"config\"."}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","plugins-service"],"pid":10664,"message":"Discovered 1 plugins."}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["info","plugins-system"],"pid":10664,"message":"Setting up [1] plugins: [translations]"}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","plugins-system"],"pid":10664,"message":"Setting up plugin \"translations\"..."}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","plugins","translations"],"pid":10664,"message":"Initializing plugin"}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["info","plugins","translations"],"pid":10664,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","root"],"pid":10664,"message":"starting root"}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","plugins-service"],"pid":10664,"message":"Plugins service starts plugins"}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["info","plugins-system"],"pid":10664,"message":"Starting [1] plugins: [translations]"}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","plugins-system"],"pid":10664,"message":"Starting plugin \"translations\"..."}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","legacy-service"],"pid":10664,"message":"starting legacy service"}
{"type":"log","@timestamp":"2019-09-12T18:35:17Z","tags":["plugin","debug"],"pid":10664,"path":"/usr/share/kibana/x-pack","message":"Found plugin at /usr/share/kibana/x-pack"}
{"type":"log","@timestamp":"2019-09-12T18:35:17Z","tags":["plugin","debug"],"pid":10664,"path":"/usr/share/kibana/src/legacy/core_plugins/apm_oss","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/apm_oss"}
{"type":"log","@timestamp":"2019-09-12T18:35:17Z","tags":["debug","root"],"pid":10664,"message":"shutting root down"}

Another thing noticed that '.security' index is also not present/available not sure whether it had been created initially when ES was enabled for security

[2019-09-13T04:21:22,099][INFO ][o.e.x.s.a.s.m.NativeRoleMappingStore] [eshost] The security index is not yet available - no role mappings can be loaded
[2019-09-13T04:21:22,100][DEBUG][o.e.x.s.a.s.m.NativeRoleMappingStore] [eshost] Security Index [.security] [exists: false] [available: false] [mapping up to date: true]
2

I doubt this has anything to do with Active Directory in Elasticsearch, but please share your kibana.yml configuration with us. Is the above all you can see in the kibana logs?

3

Yes these are the logs getting repeated

server.port: 8882
server.host: "eshost"
server.name: "eshost"
elasticsearch.hosts: ["http://eshost:9200"]
kibana.index: ".kibana"
kibana.defaultAppId: "discover"
logging.dest: /var/log/kibana.log
logging.verbose: true
xpack.security.enabled: true
xpack.security.audit.enabled: true
4

Kibana doesn't have support for AD authentication yet, so you still need to have native realm users specified in your kibana.yml as your ES username and password. Is that the case for you?

5

ok I do not know about its limitations yet. Yes I have my elasticsearch integrated with AD.
So what changes required to get the kibana also follow similar pattern so that only ELKadmins can have admin access to entire cluster, and different groups & users to have access to their respective indices where some users/groups have full control of their index and some users have only read only mode.

Do you mean the user using which the ES was integrated that user name should be mentioned in kibana.yml as well as kibana keystore? secondly I am using keystore in ES so in that case how to mention keystore in kibana.yml?

Please note - since my kibana is not up I need to make changes in config files only

6

The first answer here is still valid for 7.x regarding AD and Kibana: Kibana 5.3.0 Active Directory authentication

7

You need to setup the passwords of the built in users **, kibana user is one of them.

Once you have set the password for it, then you would add

elasticsearch.username: kibana 
elasticsearch.password: thepasswordyousethere

in kibana.yml and start it again.

** doing so will also automatically create the .security index as we discussed in the other post .

8

But these steps are for local user authentication only right? If i want to use custom user accounts and we have those in AD then how do I give every individual access to kibana dashboards? Consider I have 50 users

One more question clicked here, are all these account names mentioned in link that you posted are service accounts? and they should also be created in AD as is?

  1. If I want to use keystore then how do I mention here?
  2. In previous response by 'marius_dragomir'

When he refers 'native realm' does it mean service account name such as 'elastic' , 'kibana' , 'logstash_user' etc... or the one who is authorized to tie a knot in AD with elasticsearch like we used the account for elasticsearch present in AD as administrator?

I do have several other questions revolving around these should I ask?

10

Starting of, I think you will find our documentation very helpful. Most of your questions can be answered with information from there and this will also enhance your understanding of how and why things are set up in a certain way. You can search on www.elastic.co for any topic that you want to learn more about !

But these steps are for local user authentication only right?

These are builtin users which are kind of special local users ,not users of the native realm. Our documentation covers both in detail, please read through User authentication | Elastic Stack Overview [7.4] | Elastic

You need to configure at least one so that kibana can communicate with elasticsearch, this user is the kibana user.

If i want to use custom user accounts and we have those in AD then how do I give every individual access to kibana dashboards? Consider I have 50 users

You would need to give the users the necessary roles .read through Granting access to Kibana | Kibana Guide [8.11] | Elastic. when you have the roles ready, you can assign the roles to users in your roles.yml file that you already have

One more question clicked here, are all these account names mentioned in link that you posted are service accounts? and they should also be created in AD as is?

No, these are builtin users ,they exist in elasticsearch, you dont need to create them in AD.

If I want to use keystore then how do I mention here?

I think he meant to say reserved realm == builtin users

You can ask here or your support engineer if you have a subscription. People im this forums will try to answer when we have time, but it's on a best effort basis !

11

[root@eshost kibana]# /usr/share/kibana/bin/kibana-keystore --allow-root list
kibana
elasticsearch.username
elasticsearch.password
[root@eshost kibana]#

elasticsearch.username: "kibana"
elasticsearch.password: ${elasticsearch.password}

It did not created .security index and below are the list of indices

[root@eshost kibana]# curl -u elkadmin2 "http://192.168.1.1:9200/_cat/indices?pretty"
Enter host password for user 'elkadmin2':
green open .kibana_task_manager        5Zp52pe2T1SYKuRDaOb1xA 1 1    2 0   59.2kb  29.6kb
green open .monitoring-es-7-2019.09.11 hf7UQg01QIOG6xjO8lWcTQ 1 1   11 0  122.8kb  61.4kb
green open .monitoring-es-7-2019.09.12 FWkWRE6PTg-3jeOIl-qJuA 1 1 1036 0  725.5kb 362.7kb
green open .monitoring-es-7-2019.09.13 HGGuzXqaTDy5IcPn8Klw9Q 1 1 1440 0 1017.7kb 508.8kb
green open .monitoring-es-7-2019.09.14 yE67sngGRwyyDGou9HZ4gw 1 1 1439 0      1mb 534.8kb
green open .monitoring-es-7-2019.09.15 4lmahB9aRm2GEf-oXE_LRw 1 1    1 0    254kb 126.9kb
green open .kibana_1                   3LmFlua7SMmANjEBq94ndg 1 1    4 1   47.9kb  23.9kb
[root@eshost kibana]#

Next disabled security properties, changed from hostnames to IP address but still kibana does not comes up

{"type":"log","@timestamp":"2019-09-15T16:25:32Z","tags":["info","plugins","translations"],"pid":1855,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2019-09-15T16:25:32Z","tags":["info","plugins-system"],"pid":1855,"message":"Starting [1] plugins: [translations]"}
{"type":"log","@timestamp":"2019-09-15T16:25:36Z","tags":["fatal","root"],"pid":1855,"message":"{ ValidationError: child \"kibana\" fails because [\"kibana\" must be an object]\n    at Object.exports.process (/usr/share/kibana/node_modules/joi/lib/errors.js:196:19)\n    at internals.Object._validateWithOptions (/usr/share/kibana/node_modules/joi/lib/types/any/index.js:675:31)\n    at module.exports.internals.Any.root.validate (/usr/share/kibana/node_modules/joi/lib/index.js:146:23)\n    at Config._commit (/usr/share/kibana/src/legacy/server/config/config.js:132:34)\n    at Config.set (/usr/share/kibana/src/legacy/server/config/config.js:102:10)\n    at Config.extendSchema (/usr/share/kibana/src/legacy/server/config/config.js:74:10)\n    at extendConfigService (/usr/share/kibana/src/legacy/plugin_discovery/plugin_config/extend_config_service.js:45:10) name: 'ValidationError' }"}
{"type":"log","@timestamp":"2019-09-15T16:25:39Z","tags":["info","plugins-system"],"pid":1879,"message":"Setting up [1] plugins: [translations]"}
{"type":"log","@timestamp":"2019-09-15T16:25:39Z","tags":["info","plugins","translations"],"pid":1879,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2019-09-15T16:25:39Z","tags":["info","plugins-system"],"pid":1879,"message":"Starting [1] plugins: [translations]"}
{"type":"log","@timestamp":"2019-09-15T16:25:43Z","tags":["fatal","root"],"pid":1879,"message":"{ ValidationError: child \"kibana\" fails because [\"kibana\" must be an object]\n    at Object.exports.process (/usr/share/kibana/node_modules/joi/lib/errors.js:196:19)\n    at internals.Object._validateWithOptions (/usr/share/kibana/node_modules/joi/lib/types/any/index.js:675:31)\n    at module.exports.internals.Any.root.validate (/usr/share/kibana/node_modules/joi/lib/index.js:146:23)\n    at Config._commit (/usr/share/kibana/src/legacy/server/config/config.js:132:34)\n    at Config.set (/usr/share/kibana/src/legacy/server/config/config.js:102:10)\n    at Config.extendSchema (/usr/share/kibana/src/legacy/server/config/config.js:74:10)\n    at extendConfigService (/usr/share/kibana/src/legacy/plugin_discovery/plugin_config/extend_config_service.js:45:10) name: 'ValidationError' }"}
{"type":"log","@timestamp":"2019-09-15T16:25:46Z","tags":["info","plugins-system"],"pid":1902,"message":"Setting up [1] plugins: [translations]"}
{"type":"log","@timestamp":"2019-09-15T16:25:46Z","tags":["info","plugins","translations"],"pid":1902,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2019-09-15T16:25:46Z","tags":["info","plugins-system"],"pid":1902,"message":"Starting [1] plugins: [translations]"}
{"type":"log","@timestamp":"2019-09-15T16:25:50Z","tags":["fatal","root"],"pid":1902,"message":"{ ValidationError: child \"kibana\" fails because [\"kibana\" must be an object]\n    at Object.exports.process (/usr/share/kibana/node_modules/joi/lib/errors.js:196:19)\n    at internals.Object._validateWithOptions (/usr/share/kibana/node_modules/joi/lib/types/any/index.js:675:31)\n    at module.exports.internals.Any.root.validate (/usr/share/kibana/node_modules/joi/lib/index.js:146:23)\n    at Config._commit (/usr/share/kibana/src/legacy/server/config/config.js:132:34)\n    at Config.set (/usr/share/kibana/src/legacy/server/config/config.js:102:10)\n    at Config.extendSchema (/usr/share/kibana/src/legacy/server/config/config.js:74:10)\n    at extendConfigService (/usr/share/kibana/src/legacy/plugin_discovery/plugin_config/extend_config_service.js:45:10) name: 'ValidationError' }"}
{"type":"log","@timestamp":"2019-09-15T16:25:54Z","tags":["info","plugins-system"],"pid":1926,"message":"Setting up [1] plugins: [translations]"}
{"type":"log","@timestamp":"2019-09-15T16:25:54Z","tags":["info","plugins","translations"],"pid":1926,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2019-09-15T16:25:54Z","tags":["info","plugins-system"],"pid":1926,"message":"Starting [1] plugins: [translations]"}

[root@eshost kibana]# {"type":"log","@timestamp":"2019-09-15T16:25:58Z","tags":["fatal","root"],"pid":1926,"message":"{ ValidationError: child \"kibana\" fails because [\"kibana\" must be an object]\n    at Object.exports.process (/usr/share/kibana/node_modules/joi/lib/errors.js:196:19)\n    at internals.Object._validateWithOptions (/usr/share/kibana/node_modules/joi/lib/types/any/index.js:675:31)\n    at module.exports.internals.Any.root.validate (/usr/share/kibana/node_modules/joi/lib/index.js:146:23)\n    at Config._commit (/usr/share/kibana/src/legacy/server/config/config.js:132:34)\n    at Config.set (/usr/share/kibana/src/legacy/server/config/config.js:102:10)\n    at Config.extendSchema (/usr/share/kibana/src/legacy/server/config/config.js:74:10)\n    at extendConfigService (/usr/share/kibana/src/legacy/plugin_discovery/plugin_config/extend_config_service.js:45:10) name: 'ValidationError' }"}


current simple config without security:


server.host: "192.168.1.1"
server.name: "eshost"
elasticsearch.hosts: "http://192.168.1.1:9200"
kibana.index: ".kibana"
kibana.defaultAppId: "discover"
logging.dest: /var/log/kibana.log
12

I never said that setting a password in kibana keystore will create the security index. I said that setting the password for the built in users will.
Please do not fixate on the security index, we have established this is unrelated. I only mentioned it as something worth knowing since you were worried about it in the other thread. Let's leave this aside for now.

Did you set the passwords for the built-in users as suggested in my previous post ?

This sounds unrelated. It would be great if you could refrain from unrelated changes , while we are trying to troubleshoot an existing issue, as it is making it really hard for the folks that try to assist.

You can't have security if you disable security. We can't keep troubleshooting your security related issues if you disable security.

It fails because you added a setting in your keystore named 'kibana' which is not allowed. The error you get describes that

13

absolutely correct. I removed kibana from keystore and kibana started successfully. I noticed I was able to login via elastic user but not with kibana user (gave me error - "{"statusCode":403,"error":"Forbidden","message":"Forbidden"}" ) though the roles are not defined for both users. How come?

I used elasticsearch.username as kibana and in configuration I have used below, does this mean though kibana user is mentioned it will not be allowed to login in first instance rather to do some backend work and do more to give privileges to login as kibana user?

elasticsearch.username: kibana
elasticsearch.password: ${elasticsearch.password}
--------
[root@eshost kibana]# /usr/share/kibana/bin/kibana-keystore --allow-root list
elasticsearch.username
elasticsearch.password
[root@eshost kibana]#

yes

A question, since we have this cluster integrated with AD, for all users in AD to give them access to various dashboards, do I need to create all those AD users here in Kibana? I assume not so because it may be tedious job to create huge number of users here in Kibana. Then what is the way out?

14

kibana user is meant to be used for kibana to communicate to elasticsearch and not to be used by end users. This is expected behavior and it works as it's designed to work, no need to worry.

No, it doesn't mean that. it will work fine, no need to worry.

I replied to this in Kibana not starting for enabling AD security - #10 by ikakavas

Please read through the docs.

15

Finally all authentication issues are resolved. Many thanks to @ikakavas for guiding, assisting on the solutions. Your suggestions are helpful

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.