Access denied while opening Stack-Monitoring on Kibana 7.8

Elastic Stack Kibana
,
1

Stack Intro: I'm running ELK-Stack 7.8 with 3 Cluster-Nodes
Enabled XPack security in elasticsearch yml "xpack.security.enabled: true" and also enabled SSL/TLS on Transport and HTTP with below conf:

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/wildcard.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/wildcard.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/certs/wildcard.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/wildcard.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]

Then finally used "bin/elasticsearch-setup-passwords auto" to generate credentials for built-in users.
Configured the credentials in kibana.yml with elasticsearch.username: elastic and related password.

After all these configurations, I was able to login to Kibana dashboard using built-in user 'elastic' but ISSUE: I'm unable to view security options and getting access denied while accessing stack-monitoring with attached screenshot references

Screen Shot 2020-07-05 at 5.18.01 PM
Screen Shot 2020-07-05 at 5.18.01 PM2832×564 201 KB
Screen Shot 2020-07-05 at 5.12.03 PM
Screen Shot 2020-07-05 at 5.12.03 PM2690×1258 236 KB

2

Can you share your entire kibana.yml? I'm assuming you are not using a dedicated monitoring cluster (and instead using a single cluster to store your "production" data as well as the monitoring data"), is that correct?

3 
server.port: 5601

server.host: "localhost"

#server.basePath: ""

#server.rewriteBasePath: false

#server.maxPayloadBytes: 1048576

server.name: "<Kibana-Host-Name>"
xpack.encryptedSavedObjects.encryptionKey: "<Encryption-Key>"

elasticsearch.hosts: ["http://<ELK-Coordinator_Host-Address>:9200"]

#elasticsearch.preserveHost: true

#kibana.index: ".kibana"

#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "elastic"
elasticsearch.password: "<password>"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.

server.ssl.enabled: false
# server.ssl.certificate: /etc/kibana/certs/wildcard.crt
# server.ssl.key: /etc/kibana/certs/wildcard.key

elasticsearch.ssl.certificate: /etc/kibana/certs/wildcard.crt
elasticsearch.ssl.key: /etc/kibana/certs/wildcard.key

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca.crt" ]

#elasticsearch.ssl.verificationMode: full

#elasticsearch.pingTimeout: 1500

#elasticsearch.requestTimeout: 30000

#elasticsearch.requestHeadersWhitelist: [ authorization ]

#elasticsearch.customHeaders: {}

#elasticsearch.shardTimeout: 30000

#elasticsearch.startupTimeout: 5000

#elasticsearch.logQueries: false

#pid.file: /var/run/kibana.pid

#logging.dest: stdout

#logging.silent: false

#logging.quiet: false

#logging.verbose: false

#ops.interval: 5000

#i18n.locale: "en"
4

Thanks @chrisronline for your response.
Attached the full kibana.yml contents.
I'm not using dedicated monitoring node.

5

Thanks for that.

Are you seeing any errors/warnings in the Kibana server log?

6 

{"type":"response","@timestamp":"2020-07-06T15:03:56Z","tags":[],"pid":7759,"method":"post","statusCode":403,"req":{"url":"/api/monitoring/v1/clusters","method":"post","headers":{"host":"<Kibana-Server-IP>","x-real-ip":"<user-machine>","x-forwarded-for":"<user-machine>","x-forwarded-proto":"https","connection":"close","content-length":"101","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","content-type":"application/json;charset=utf-8","kbn-version":"7.8.0","origin":"https://<Kibana-Server-IP>","referer":"https://<Kibana-Server-IP>/app/monitoring"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://<Kibana-Server-IP>/app/monitoring"},"res":{"statusCode":403,"responseTime":61,"contentLength":9},"message":"POST /api/monitoring/v1/clusters 403 61ms - 9.0B"}


{"type":"response","@timestamp":"2020-07-06T15:03:56Z","tags":[],"pid":7759,"method":"get","statusCode":200,"req":{"url":"/api/monitoring/v1/check_access","method":"get","headers":{"host":"<Kibana-Server-IP>","x-real-ip":"<user-machine>","x-forwarded-for":"<user-machine>","x-forwarded-proto":"https","connection":"close","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","kbn-version":"7.8.0","referer":"https://<Kibana-Server-IP>/app/monitoring"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://<Kibana-Server-IP>/app/monitoring"},"res":{"statusCode":200,"responseTime":26,"contentLength":9},"message":"GET /api/monitoring/v1/check_access 200 26ms - 9.0B"}


{"type":"log","@timestamp":"2020-07-06T15:03:56Z","tags":["error","plugins","monitoring","monitoring"],"pid":7759,"message":"{ Error: [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [elastic] (and) [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [elastic]\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)\n  status: 403,\n  displayName: 'AuthorizationException',\n  message:\n   '[security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [elastic] (and) [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [elastic]',\n  path:\n   '/*%3A.monitoring-es-6-*%2C*%3A.monitoring-es-7-*%2C.monitoring-es-6-*%2C.monitoring-es-7-*/_search',\n  query:\n   { size: 10000,\n     ignore_unavailable: true,\n     filter_path:\n      'hits.hits._index,hits.hits._source.cluster_uuid,hits.hits._source.cluster_name,hits.hits._source.version,hits.hits._source.license.status,hits.hits._source.license.type,hits.hits._source.license.issue_date,hits.hits._source.license.expiry_date,hits.hits._source.license.expiry_date_in_millis,hits.hits._source.cluster_stats,hits.hits._source.cluster_state,hits.hits._source.cluster_settings.cluster.metadata.display_name' },\n  body:\n   { error:\n      { root_cause: [Array],\n        type: 'search_phase_execution_exception',\n        reason: 'all shards failed',\n        phase: 'can_match',\n        grouped: true,\n        failed_shards: [Array] },\n     status: 403 },\n  statusCode: 403,\n  response:\n   '{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/read/search[can_match]] is unauthorized for user [elastic]\"},{\"type\":\"security_exception\",\"reason\":\"action [indices:data/read/search[can_match]] is unauthorized for user [elastic]\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"can_match\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\".monitoring-es-7-2020.07.05\",\"node\":\"vWNN1kzSQzaGR6M9BvY4lw\",\"reason\":{\"type\":\"security_exception\",\"reason\":\"action [indices:data/read/search[can_match]] is unauthorized for user [elastic]\",\"caused_by\":{\"type\":\"illegal_state_exception\",\"reason\":\"There are no external requests known to support wildcards that don\\'t support replacing their indices\"}}},{\"shard\":0,\"index\":\".monitoring-es-7-2020.07.06\",\"node\":\"93kgxcPJTyWU1mttNFNlug\",\"reason\":{\"type\":\"security_exception\",\"reason\":\"action [indices:data/read/search[can_match]] is unauthorized for user [elastic]\",\"caused_by\":{\"type\":\"illegal_state_exception\",\"reason\":\"There are no external requests known to support wildcards that don\\'t support replacing their indices\"}}}]},\"status\":403}',\n  toString: [Function],\n  toJSON: [Function] }"}


{"type":"response","@timestamp":"2020-07-06T15:03:56Z","tags":[],"pid":7759,"method":"post","statusCode":403,"req":{"url":"/api/monitoring/v1/clusters","method":"post","headers":{"host":"<Kibana-Server-IP>","x-real-ip":"<user-machine>","x-forwarded-for":"","x-forwarded-proto":"https","connection":"close","content-length":"101","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","content-type":"application/json;charset=utf-8","kbn-version":"7.8.0","origin":"https://<Kibana-Server-IP>","referer":"https://<Kibana-Server-IP>/app/monitoring"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://<Kibana-Server-IP>/app/monitoring"},"res":{"statusCode":403,"responseTime":63,"contentLength":9},"message":"POST /api/monitoring/v1/clusters 403 63ms - 9.0B"}

@chrisronline Above is the kibana-log

7

Issue Resolved. There was a typo in one of the ELK-Cluster Node yml conf. file
Rectifying and fixing it resolved the issue.

I'm able to access stack-monitoring service successfully.

Thank You @chrisronline

8

Summary: xpack.security.enabled: true should be enabled on ALL elasticsearch nodes

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.