"You are not authorized to access Monitoring" after upgrade to 8.1.0

[bricevalenza]

Hello,

after upgrading my stack to version 8.1.0, I can't access the monitoring page on Kibana.

I checked rights on my (local) account and it's OK. I also made a role with very large accesses, and still the same

I can read in the discover page the events present in the .monitoring indexes.

does anyone have an idea?

[rashmi (kulkarni)]

What does the Kibana and ES log say ? Can you share your entire kibana.yml? I'm assuming you are not using a dedicated monitoring cluster (and instead using a single cluster to store your "production" data as well as the monitoring data"), is that correct?

cc @rhodesjason any inputs?

[bricevalenza]

Hi,

yes, I only have one cluster with production and monitoring

I have no particular log on Elasticsearch nodes

On Kibana :

Mar 15 10:35:02 S400LIAKIBANA1 kibana: [2022-03-15T10:35:02.620+01:00][ERROR][plugins.monitoring.monitoring] ResponseError: security_exception: [security_exception] Reason: action [indice
s:data/read/search] is unauthorized for user [brice.valenza] with roles [superuser,kibana_admin,monitoring_user,eol_admin,kibana_system,kibana_user], this action is granted by the index p
rivileges [read,all]
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at runMicrotasks (<anonymous>)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at processTicksAndRejections (node:internal/process/task_queues:96:5)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at KibanaTransport.request (/usr/share/kibana/src/core/server/elasticsearch/client/create_transport.js:64:16)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at Client.SearchApi [as search] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/search.js:65:12)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at /usr/share/kibana/x-pack/plugins/monitoring/server/static_globals.js:56:11
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at mbSafeQuery (/usr/share/kibana/x-pack/plugins/monitoring/server/lib/mb_safe_query.js:31:12)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at Object.getLegacyClusterShim (/usr/share/kibana/x-pack/plugins/monitoring/server/static_globals.js:46:70)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at callWithRequest (/usr/share/kibana/x-pack/plugins/monitoring/server/plugin.js:352:30)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at getClustersFromRequest (/usr/share/kibana/x-pack/plugins/monitoring/server/lib/cluster/get_clusters_from_request.js:75:16)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at Object.handler (/usr/share/kibana/x-pack/plugins/monitoring/server/routes/api/v1/cluster/clusters.js:56:20)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at handler (/usr/share/kibana/x-pack/plugins/monitoring/server/plugin.js:361:28)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
Mar 15 10:35:02 S400LIAKIBANA1 kibana: at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)

The kibana configuration

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

server.host: "HOSTNAMEKIBANA"


server.name: "HOSTNAMEKIBANA"


elasticsearch.hosts:
  - "http://IP1:9200"
  - "http://IP2:9200"


# When this setting's value is true Kibana uses the hostname specified in the server.h
elasticsearch.username: "USERKIBANA"
elasticsearch.password: "PASSWORDKIBANA"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/CRTNAME.crt
server.ssl.key: /etc/kibana/certs/KEYNAME.key

xpack.security.encryptionKey: "ENCRYPTIONKEY"
elasticsearch.requestTimeout: 120000
logging.root.level: error
xpack.monitoring.kibana.collection.enabled: false

Thanks

[haukebruno]

We experience the same issue but with 8.0.0 (this was a fresh install, no clue what happened than).

For trial&error reasons I gave my user at least every single role available, but didn't work out.

ES logs are clean, Kibana log throws:

[2022-03-16T18:29:56.233+00:00][ERROR][plugins.monitoring.monitoring] ResponseError: security_exception: [security_exception] Reason: action [indices:data/read/search] is unauthorized for user [hauke] with roles [superuser,kibana_admin,monitoring_user,editor,machine_learning_user,remote_monitoring_collector,snapshot_user,transport_client,watcher_user,beats_admin,ingest_admin,logstash_admin,machine_learning_admin,rollup_admin,transform_admin,watcher_admin,apm_system,beats_system,kibana_system,logstash_system,viewer,transform_user,enrich_user,remote_monitoring_agent,rollup_user], this action is granted by the index privileges [read,all]

Monitoring indices are indeed there and get also new docs.

Edit: We are also on only one cluster, no dedicate monitoring cluster.

[haukebruno]

Ok, checking the Github issues helped (at least me) out

Improved error logging - include caused_by.reason

After adding the remote_cluster_client role to our second ES node (had only data, ingest before) the monitoring stuff started to work again.

[Lagendijkr]

Kibana: Error description
security_exception: [security_exception] Reason: action [indices:data/read/search] is unauthorized for service account [elastic/kibana], this action is granted by the index privileges [read,all]
or
security_exception: [security_exception] Reason: action [indices:data/read/search] is unauthorized for user [elastic] with roles [superuser], this action is granted by the index privileges [read,all]

Kibana > Dev Tools: Query DSL
POST *:.monitoring-*/_search

Returns detailed thrown exception
node [$nodename] does not have the remote cluster client role enabled

Solution
Proper configuration of Node roles within cluster resolved the problem.
In my particular case the adding the role: remote_cluster_client.

Addendum;
Documenting this particular sequence for future reference on here.