Kibana not showing data after turning on xpack security and TLS

Elastic Stack Kibana
1

Yesterday I updated the security on our small 3 node cluster of Elasticsearch 7.17.5 running on CentOS 7. I did my best to follow the documentation: I added the passwords for the built in accounts, set up the kibana_keystore and used elasticsearch-certutil and set up TLS encryption.

The cluster came back up and to me (a non-user called in periodically to research and fix issues others notice) it looks like the cluster is up and healthy. I can log into Kibana and all seems fine. But the indexes aren't getting data. When I use discover, I can see index data coming in right up until the time I enabled security

We are forwarding the data from AWS and the parameter store has been updated with the credentials for the kibana_system user. We have nginx set up on the front end and it sends elasticsearch traffic to the elasticsearch port and kibana traffic to the kibana port. If I look in the nginx logs, I see elasticsearch data being sent through.

looking in the elasticsearch logs I see entries like these:

[2022-10-26T04:49:44,617][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-10-5-195.XXXX.com] successfully reloaded changed geoip database file [/tmp/elasticsearch-7703106640032577397/geoip-databases/mCBvIjHNTUufxZKjgAfHzw/GeoLite2-Country.mmdb]
[2022-10-26T04:49:48,212][INFO ][o.e.i.g.DatabaseNodeService] [ip-10-10-5-195.XXXX.com] successfully reloaded changed geoip database file [/tmp/elasticsearch-7703106640032577397/geoip-databases/mCBvIjHNTUufxZKjgAfHzw/GeoLite2-City.mmdb]
[2022-10-26T04:50:03,853][INFO ][o.e.t.LoggingTaskListener] [ip-10-10-5-195.XXXX.com] 195 finished with response BulkByScrollResponse[took=113ms,timed_out=false,sliceId=null,updated=18,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=,search_failures=]
[2022-10-26T04:50:04,612][INFO ][o.e.t.LoggingTaskListener] [ip-10-10-5-195.XXXX.com] 206 finished with response BulkByScrollResponse[took=741.5ms,timed_out=false,sliceId=null,updated=546,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=,search_failures=]
[2022-10-26T05:30:59,773][INFO ][o.e.t.LoggingTaskListener] [ip-10-10-5-195.XXXX.com] 31065 finished with response BulkByScrollResponse[took=114.1ms,timed_out=false,sliceId=null,updated=18,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=,search_failures=]
[2022-10-26T05:31:00,034][INFO ][o.e.t.LoggingTaskListener] [ip-10-10-5-195.XXXX.com] 31075 finished with response BulkByScrollResponse[took=319.4ms,timed_out=false,sliceId=null,updated=552,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=,search_failures=]

...which don't look too sinister - but I don't know how to interpret these. Not sure if they indicate successes or not.

Can any of y'all give me some thoughts on how to troubleshoot this and get the indexes showing up in Kibana again? I hesitate to rebuild the indexes - but can if req'd. Did I miss a step somewhere?

2

Looking through the elasticsearch logs, after a restart of the service, I see this:

[2022-10-26T06:19:39,233][INFO ][o.e.x.s.a.RealmsAuthenticator] [ip-10-10-5-195.XXXX.com] Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana]

I look through kibana.yml and the elasticsearch.username is kibana_system - so I'm not sure where the kibana user is coming from - but this is an INFO level log entry. Not sure what to make of this.

3

The kibana_system user is designed for Kibana access only and it's likely that the source it unable to upload data due to incorrect permissions.

You should create a new user with ingest level permissions.

4

It's late here - I've created the user in stack management in kibana and updated the AWS parameter store and also nginx to let it through - is there anywhere else I need to add this user?

are there other roles I should add beyond ingest_manager to make this work?

I'm not seeing any new entries into the indexes yet.

5

Added some further permissions beyond ingest_manager to the new account and logs started showing up in Kibana. Thank you for your help!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.