Cannot authenticate with kibana user to the stack after I enabled security 7.7.1

elasticsearch.yml sec setup.


xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

here is my kibana.yml -sending only the uncommented lines

server.port: 5601

server.name: "infralogs-elasticsearchm-101w"

elasticsearch.hosts: ["http://infralogs-elasticsearchm-101w.active.tan:9200"]

# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana"  < i uncomment these when I try security 
#elasticsearch.password: "31kibana" < i uncomment these when I try security 

pid.file: /opt/kibana-7.7.1-linux-x86_64/kibana.pid

logging.dest: /opt/kibana-7.7.1-linux-x86_64/logs/stdou

**Do I need to put these in when security is enabled?**

#xpack.security.enabled: true
#xpack.security.transport.ssl.enabled: true
#xpack.security.transport.ssl.verification_mode: certificate
#xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
#xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12

here is some log snipits showing problem authenticating after restarting kibana

system"],"pid":53551,"message":"Setting up [76] plugins: [taskManager,siem,licensing,eventLog,encryptedSavedObjects,code,visTypeVega,usageCollection,metrics,ossTelemetry,lens,telemetryCollectionManager,telemetry,telemetryCollectionXpack,timelion,features,kibanaLegacy,devTools,apm_oss,translations,rollup,observability,uiActions,statusPage,share,savedObjects,newsfeed,kibanaUtils,kibanaReact,inspector,maps,embeddable,drilldowns,advancedUiActions,esUiShared,discover,charts,bfetch,expressions,visualizations,data,home,cloud,console,consoleExtensions,searchprofiler,painlessLab,canvas,management,upgradeAssistant,security,snapshotRestore,transform,licenseManagement,indexManagement,remoteClusters,watcher,reporting,advancedSettings,spaces,actions,case,alerting,alertingBuiltins,triggers_actions_ui,apm,uptime,ml,telemetryManagementSection,file_upload,dataEnhanced,infra,monitoring,navigation,graph,dashboard]"}
{"type":"log","@timestamp":"2020-11-04T15:11:07Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":53551,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-11-04T15:11:08Z","tags":["warning","plugins","security","config"],"pid":53551,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-11-04T15:11:08Z","tags":["warning","plugins","security","config"],"pid":53551,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2020-11-04T15:11:08Z","tags":["warning","plugins","actions","actions"],"pid":53551,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."}
{"type":"log","@timestamp":"2020-11-04T15:11:08Z","tags":["warning","plugins","alerting","plugins","alerting"],"pid":53551,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."}
{"type":"log","@timestamp":"2020-11-04T15:11:08Z","tags":["info","plugins","monitoring","monitoring"],"pid":53551,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2020-11-04T15:11:08Z","tags":["warning","plugins","monitoring","monitoring"],"pid":53551,"message":"X-Pack Monitoring Cluster Alerts will not be available: undefined"}
{"type":"log","@timestamp":"2020-11-04T15:11:08Z","tags":["warning","plugins","licensing"],"pid":53551,"message":"License information could not be obtained from Elasticsearch due to [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"failed to authenticate user [kibana]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"failed to authenticate user [kibana]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"} error"}

{"type":"log","@timestamp":"2020-11-04T15:11:08Z","tags":["info","plugins","searchprofiler"],"pid":53551,"message":"You cannot use searchprofiler 
...because license information is not available at this time."}
...
...
["info","plugins","watcher"],"pid":53551,"message":"You cannot use watcher because license information is not available at this time."}
{"type":"log","@timestamp":"2020-11-04T15:11:08Z","tags":
{"type":"log","@timestamp":"2020-11-04T15:11:08Z","tags":["error","savedobjects-service"],"pid":53551,"message":"Unable to retrieve version information from Elasticsearch nodes."}


{"type":"log","@timestamp":"2020-11-04T15:13:45Z","tags":["info","plugins-system"],"pid":53551,"message":"Stopping all plugins."}

{"type":"log","@timestamp":"2020-11-04T15:14:02Z","tags":["info","plugins-system"],"pid":53892,"message":"Setting up [76] plugins: [taskManager,.........,dashboard]"}
{"type":"log","@timestamp":"2020-11-04T15:14:02Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":53892,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-11-04T15:14:02Z","tags":["warning","plugins","security","config"],"pid":53892,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-11-04T15:14:02Z","tags":["warning","plugins","security","config"],"pid":53892,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2020-11-04T15:14:02Z","tags":["warning","plugins","actions","actions"],"pid":53892,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."}
{"type":"log","@timestamp":"2020-11-04T15:14:02Z","tags":["warning","plugins","alerting","plugins","alerting"],"pid":53892,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."}
{"type":"log","@timestamp":"2020-11-04T15:14:02Z","tags":
{"type":"log","@timestamp":"2020-11-04T15:14:02Z","tags":["warning","plugins","licensing"],"pid":53892,"message":"License information could not be obtained from Elasticsearch due to [security_exception] missing authentication credentials for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication credentials for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication credentials for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"} error"}
{"type":"log","@timestamp":"2020-11-04T15:14:02Z","tags":["info","savedobjects-service"],"pid":53892,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}

{"type":"log","@timestamp":"2020-11-04T15:15:17Z","tags":["error","elasticsearch","admin"],"pid":53892,"message":"Request error, retrying\nGET http://infralogs-elasticsearchm-101w.active.tan:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => connect ECONNREFUSED 10.209.3.11:9200"}
{"type":"log","@timestamp":"2020-11-04T15:15:17Z","tags":["warning","elasticsearch","admin"],"pid":53892,"message":"Unable to revive connection: http://infralogs-elasticsearchm-101w.active.tan:9200/"}
{"type":"log","@timestamp":"2020-11-04T15:15:17Z","tags":["warning","elasticsearch","admin"],"pid":53892,"message":"No living connections"}

{"type":"log","@timestamp":"2020-11-04T15:15:30Z","tags":["info","savedobjects-service"],"pid":53892,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2020-11-04T15:15:30Z","tags":["info","savedobjects-service"],"pid":53892,"message":"Creating index .kibana_task_manager_1."}
{"type":"log","@timestamp":"2020-11-04T15:15:30Z","tags":["info","savedobjects-service"],"pid":53892,"message":"Creating index .kibana_1."}
{"type":"log","@timestamp":"2020-11-04T15:15:32Z","tags":["info","plugins","searchprofiler"],"pid":53892,"message":"You cannot use searchprofiler because license information is not available at this time."}
...
...
["info","plugins","watcher"],"pid":53892,"message":"You cannot use watcher because license information is not available at this time."}
{"type":"log","@timestamp":"2020-11-04T15:15:32Z","tags":["info","plugins","monitoring","monitoring","kibana-system"],"pid":53892,"message":"Starting [53] plugins: [taskManager,siem,licensing,eventLog,encryptedSavedObjects,code,visTypeVega,usageCollection,metrics,ossTelemetry,lens,telemetryCollectionManager,telemetry,telemetryCollectionXpack,timelion,features,kibanaLegacy,apm_oss,translations,rollup,share,bfetch,expressions,visualizations,data,home,cloud,console,consoleExtensions,searchprofiler,painlessLab,canvas,upgradeAssistant,security,snapshotRestore,transform,licenseManagement,indexManagement,remoteClusters,watcher,spaces,actions,case,alerting,apm,alertingBuiltins,uptime,ml,file_upload,dataEnhanced,infra,monitoring,graph]"}
{"type":"log","@timestamp":"2020-11-04T15:16:58Z","tags":["status","plugin:kibana@7.7.1","info"],"pid":53892,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-11-04T15:16:58Z","tags":["info","plugins","taskManager","taskManager"],"pid":53892,"message":"TaskManager is identified by the Kibana UUID: 53fbaab4-d6dc-4484-b2e0-aff006ce481e"}
{"type":"log","@timestamp":"2020-11-04T15:16:58Z","tags":["status","plugin:elasticsearch@7.7.1","info"],"pid":53892,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2020-11-04T15:16:58Z","tags":["status","plugin:elasticsearch@7.7.1","info"],"pid":53892,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

here is what I get when I try to use the command - iKNOW I used this password when I did the setup. how can I fix that? I understand I cannot run the setup again.

[root@infralogs-elasticsearchm-101w ~]# curl http://infralogs-elasticsearchm-101w.active.tan:9200/_security/_authenticate -u kibana:31kibana
{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [kibana]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"failed to authenticate user [kibana]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
curl http://infralogs-elasticsearchm-101w.active.tan:9200/_security/_authenticate -u kibana:31kibana

Do you still have access to Elasticsearch via the default elastic user? If so, use that to reset the password of the kibana user.

curl http://infralogs-elasticsearchm-101w.active.tan:9200/_security/_authenticate -u elastic:31astic
{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

could it be my certs need to be redone? I used to have 5 nodes in the data stack but that was changed to only 3 nodes. I use the same cert on all hosts.

used to have 2 master, 5 data,3 logstash

now

2 master , 3 data, 3 logstash

It sounds like you need to check your Elasticsearch logs.
There are many possible causes for failing to authenticate a user, and in almost every case you need to look at the logs to distinguish between them (we do not expose the internal workings of the cluster to users that were not authenticated)

topic can be closed -we ripped out all I did for first install and started from scratch including new certs and new internal password set up and I worked.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.