Kibana: Configure Security - missing authentication credentials for REST request

Elastic Stack Kibana
1

Good day,

I'm curious about the alert-functionality in Kibana. Therefore I'm setting up the security in Kibana.

I already set the elasticsearch passwords (interactive).

This are my configs

kibana.yml

server.name: kibana
server.host: "0"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
xpack.security.enabled: true
elasticsearch.username: "kibana_system"
elasticsearch.password: <my-kibana_system-password>
xpack.security.encryptionKey: <my-32-char-key>
xpack.security.session.idleTimeout: "1h"
xpack.security.session.lifespan: "30d"

This is my error log from kibana:

{"type":"log","@timestamp":"2020-11-26T14:55:11Z","tags":["warning","savedobjects-service"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."}
{"type":"log","@timestamp":"2020-11-26T14:55:11Z","tags":["warning","savedobjects-service"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."}
{"type":"log","@timestamp":"2020-11-26T14:55:27Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"auditTrail\" is disabled."}
{"type":"log","@timestamp":"2020-11-26T14:55:27Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2020-11-26T14:55:27Z","tags":["warning","config","deprecation"],"pid":6,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2020-11-26T14:55:28Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [96] plugins: [usageCollection,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,xpackLegacy,securityOss,newsfeed,mapsLegacy,kibanaLegacy,taskManager,licensing,globalSearch,globalSearchProviders,code,share,legacyExport,embeddable,uiActionsEnhanced,expressions,data,home,console,consoleExtensions,apmOss,observability,cloud,management,indexPatternManagement,advancedSettings,searchprofiler,painlessLab,grokdebugger,savedObjects,dashboard,visualizations,visTypeVega,visTypeTimelion,timelion,features,upgradeAssistant,security,snapshotRestore,encryptedSavedObjects,ingestManager,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,dashboardMode,beatsManagement,transform,ingestPipelines,licenseManagement,graph,dataEnhanced,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,fileUpload,maps,esUiShared,charts,visTypeVislib,visTypeTimeseries,rollup,visTypeTagcloud,visTypeMetric,lens,discover,discoverEnhanced,savedObjectsManagement,spaces,reporting,lists,eventLog,actions,case,alerts,stackAlerts,triggersActionsUi,ml,securitySolution,infra,monitoring,logstash,apm,uptime,watcher,bfetch,canvas,translations]"}
{"type":"log","@timestamp":"2020-11-26T14:55:28Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-11-26T14:55:28Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2020-11-26T14:55:29Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":6,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-11-26T14:55:29Z","tags":["warning","plugins","ingestManager"],"pid":6,"message":"Fleet APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."}
{"type":"log","@timestamp":"2020-11-26T14:55:29Z","tags":["warning","plugins","reporting","config"],"pid":6,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-11-26T14:55:29Z","tags":["warning","plugins","reporting","config"],"pid":6,"message":"Found 'server.host: \"0\"' in Kibana configuration. This is incompatible with Reporting. To enable Reporting to work, 'xpack.reporting.kibanaServer.hostname: 0.0.0.0' is being automatically to the configuration. You can change the setting to 'server.host: 0.0.0.0' or add 'xpack.reporting.kibanaServer.hostname: 0.0.0.0' in kibana.yml to prevent this message."}
{"type":"log","@timestamp":"2020-11-26T14:55:29Z","tags":["warning","plugins","reporting","config"],"pid":6,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 8.2.2004 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2020-11-26T14:55:29Z","tags":["warning","plugins","actions","actions"],"pid":6,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."}
{"type":"log","@timestamp":"2020-11-26T14:55:29Z","tags":["warning","plugins","alerts","plugins","alerting"],"pid":6,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."}
{"type":"log","@timestamp":"2020-11-26T14:55:29Z","tags":["info","plugins","monitoring","monitoring"],"pid":6,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2020-11-26T14:55:29Z","tags":["info","savedobjects-service"],"pid":6,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-11-26T14:55:29Z","tags":["error","elasticsearch","data"],"pid":6,"message":"[security_exception]: missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}
{"type":"log","@timestamp":"2020-11-26T14:55:29Z","tags":["error","savedobjects-service"],"pid":6,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2020-11-26T14:55:29Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to [security_exception] missing authentication credentials for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication credentials for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication credentials for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"} error"}
{"type":"log","@timestamp":"2020-11-26T14:55:30Z","tags":["warning","plugins","monitoring","monitoring"],"pid":6,"message":"X-Pack Monitoring Cluster Alerts will not be available: [security_exception] missing authentication credentials for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }"}
{"type":"log","@timestamp":"2020-11-26T14:55:32Z","tags":["error","elasticsearch","data"],"pid":6,"message":"[security_exception]: missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}
{"type":"log","@timestamp":"2020-11-26T14:55:34Z","tags":["error","elasticsearch","data"],"pid":6,"message":"[security_exception]: missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}

my versions:
kibana:7.10.0
elasticsearch:7.10.0

Everything runs in docker. Logstash is working fine with the 'elastic'-user and 'elastic'-password.

I followed the instructions under: https://www.elastic.co/guide/en/kibana/current/using-kibana-with-security.html

But I did not follow through the optional steps:
5 Optional: Configure Kibana to encrypt communications.
6 Optional: Configure Kibana to authenticate to Elasticsearch with a client certificate.

Any help would be much appreciated.

2

I checked the user name and the password with:

'curl --user kibana_system -X GET "http://elasticsearch:9200/?pretty" '

to know that it's correct.

Enter host password for user 'kibana_system':
{
  "name" : "b3a844564b02",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "64Tw7z-4SyaF6VQ2gFITzw",
  "version" : {
    "number" : "7.10.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "51e9d6f22758d0374a0f3f5c6e8f3a7997850f96",
    "build_date" : "2020-11-09T21:30:33.964949Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
3

This is my elasticsearch.yml:

cluster.name: "docker-cluster"
network.host: 0.0.0.0
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/elastic-certificates.p12

xpack.security.transport.ssl.keystore.password: <pw>
xpack.security.transport.ssl.truststore.password: <pw>
discovery.type: single-node
4

Given that you're getting this error in your log, I would double-check that:

  1. Your key is surrounded by quotes, and
  2. Your key is 32 characters long.

The code that checks for the key is here, if that helps debug the key exists when you start Kibana.

5

Thank you @clintandrewhall for your support:

I looked again and the encryptionKey looks fine.

This is my current kibana.yml:

server.name: kibana
server.host: "0"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
elasticsearch.username: "kibana_system"
elasticsearch.password: "<pw>"
xpack.security.enabled: true
xpack.security.encryptionKey: "tG0HPMzDMdwXRbhP4i1r7k9vvaqfMLAR"
xpack.security.session.idleTimeout: "1h"
xpack.security.session.lifespan: "30d"

This is my current kibana (error) log with the given config:

{"type":"log","@timestamp":"2020-11-27T16:24:52Z","tags":["warning","savedobjects-service"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."}
{"type":"log","@timestamp":"2020-11-27T16:24:52Z","tags":["warning","savedobjects-service"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."}
{"type":"log","@timestamp":"2020-11-27T16:25:04Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2020-11-27T16:25:04Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"auditTrail\" is disabled."}
{"type":"log","@timestamp":"2020-11-27T16:25:04Z","tags":["warning","config","deprecation"],"pid":6,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2020-11-27T16:25:04Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [96] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,expressions,data,home,observability,cloud,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,dashboard,visualizations,visTypeVega,visTypeTimelion,timelion,features,upgradeAssistant,security,snapshotRestore,enterpriseSearch,encryptedSavedObjects,ingestManager,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboardMode,beatsManagement,transform,ingestPipelines,maps,licenseManagement,graph,dataEnhanced,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,esUiShared,charts,lens,visTypeVislib,visTypeTimeseries,rollup,visTypeTagcloud,visTypeMetric,watcher,discover,discoverEnhanced,savedObjectsManagement,spaces,reporting,lists,eventLog,actions,case,alerts,stackAlerts,triggersActionsUi,ml,securitySolution,infra,monitoring,logstash,apm,uptime,bfetch,canvas]"}
{"type":"log","@timestamp":"2020-11-27T16:25:05Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-11-27T16:25:05Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2020-11-27T16:25:05Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":6,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-11-27T16:25:05Z","tags":["warning","plugins","ingestManager"],"pid":6,"message":"Fleet APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."}
{"type":"log","@timestamp":"2020-11-27T16:25:05Z","tags":["warning","plugins","reporting","config"],"pid":6,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-11-27T16:25:05Z","tags":["warning","plugins","reporting","config"],"pid":6,"message":"Found 'server.host: \"0\"' in Kibana configuration. This is incompatible with Reporting. To enable Reporting to work, 'xpack.reporting.kibanaServer.hostname: 0.0.0.0' is being automatically to the configuration. You can change the setting to 'server.host: 0.0.0.0' or add 'xpack.reporting.kibanaServer.hostname: 0.0.0.0' in kibana.yml to prevent this message."}
{"type":"log","@timestamp":"2020-11-27T16:25:05Z","tags":["warning","plugins","reporting","config"],"pid":6,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 8.2.2004 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2020-11-27T16:25:05Z","tags":["warning","plugins","actions","actions"],"pid":6,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."}
{"type":"log","@timestamp":"2020-11-27T16:25:05Z","tags":["warning","plugins","alerts","plugins","alerting"],"pid":6,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."}
{"type":"log","@timestamp":"2020-11-27T16:25:05Z","tags":["info","plugins","monitoring","monitoring"],"pid":6,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2020-11-27T16:25:05Z","tags":["info","savedobjects-service"],"pid":6,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-11-27T16:25:06Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to [security_exception] missing authentication credentials for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication credentials for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication credentials for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"} error"}
{"type":"log","@timestamp":"2020-11-27T16:25:06Z","tags":["warning","plugins","monitoring","monitoring"],"pid":6,"message":"X-Pack Monitoring Cluster Alerts will not be available: [security_exception] missing authentication credentials for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }"}
{"type":"log","@timestamp":"2020-11-27T16:25:06Z","tags":["error","elasticsearch","data"],"pid":6,"message":"[security_exception]: missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}
{"type":"log","@timestamp":"2020-11-27T16:25:06Z","tags":["error","savedobjects-service"],"pid":6,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2020-11-27T16:25:08Z","tags":["error","elasticsearch","data"],"pid":6,"message":"[security_exception]: missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}
{"type":"log","@timestamp":"2020-11-27T16:25:11Z","tags":["error","elasticsearch","data"],"pid":6,"message":"[security_exception]: missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}
{"type":"log","@timestamp":"2020-11-27T16:25:13Z","tags":["error","elasticsearch","data"],"pid":6,"message":"[security_exception]: missing authentication credentials for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]"}

But you are right. It seems odd, that kibana complains over an unset encryptionKey although it is set.

6

Something to try would be to ensure the kibana.yml file you're using is actually the one being loaded. Try changing some key/value to something completely incompatible, (e.g. add foo: "bar" to the end, or set xpack.security.enabled: "jurilz") and see if the incompatibility is also logged.

7

I added foo: "bar" at the end of the kibana.yml but I didn't find any foo or bar in the logs.

How can I ensure that my kibana.yml is beeing loaded?

8

@clintandrewhall Thank you very much. Your advice did the trick.

The error was that my kibana.yml wasn't in the config-directory.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.