Monitoring user

[sredne]

Hi all,

I'm still setting up my cluster with x-pack monitoring and Kibana. I setup the "update-monitor" user to have the kibana_user, monitoring_user and reporting_user role and set that as the login account for Kibana. When I try to view the monitoring page in Kibana I get an Access is Denied message. In the Kibana logs, I keep getting the following error. I'm not sure why it wouldn't have permissions. Any ideas?

  log   [15:29:09.862] [error][monitoring-ui] [security_exception] action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [update-monitor] :: {"path":"/_xpack/monitoring/_bulk","query":{"system_id":"kibana","system_api_version":"2","interval":"10000ms"},"body":"{\"index\":{\"_type\":\"kibana_stats\"}}\n{\"kibana\":{\"uuid\":\"498f0d7c-6342-4ce7-aa1b-b4b2d7e5df3e\",\"name\":\"a01fpupelaec001\",\"host\":\"a01fpupelaec001\",\"transport_address\":\"a01fpupelaec001:5601\",\"version\":\"5.2.0\",\"snapshot\":false,\"status\":\"green\"},\"concurrent_connections\":2,\"os\":{\"load\":{\"1m\":0,\"5m\":0,\"15m\":0},\"memory\":{\"total_in_bytes\":4294496256,\"free_in_bytes\":319434752,\"used_in_bytes\":3975061504},\"uptime_in_millis\":83469211.8564},\"process\":{\"event_loop_delay\":2.084201008081436,\"memory\":{\"heap\":{\"total_in_bytes\":71065600,\"used_in_bytes\":59825972,\"size_limit\":750780416},\"resident_set_size_in_bytes\":87064576},\"uptime_in_millis\":533268},\"requests\":{\"disconnects\":0,\"total\":1,\"status_codes\":{\"200\":1}},\"response_times\":{\"average\":17,\"max\":17},\"timestamp\":\"2017-02-03T15:29:08.500Z\"}\n{\"index\":{\"_index\":\"_data\",\"_type\":\"kibana\",\"_id\":\"498f0d7c-6342-4ce7-aa1b-b4b2d7e5df3e\"}}\n{\"kibana\":{\"uuid\":\"498f0d7c-6342-4ce7-aa1b-b4b2d7e5df3e\",\"name\":\"a01fpupelaec001\",\"host\":\"a01fpupelaec001\",\"transport_address\":\"a01fpupelaec001:5601\",\"version\":\"5.2.0\",\"snapshot\":false,\"status\":\"green\"},\"concurrent_connections\":2,\"os\":{\"load\":{\"1m\":0,\"5m\":0,\"15m\":0},\"memory\":{\"total_in_bytes\":4294496256,\"free_in_bytes\":319434752,\"used_in_bytes\":3975061504},\"uptime_in_millis\":83469211.8564},\"process\":{\"event_loop_delay\":2.084201008081436,\"memory\":{\"heap\":{\"total_in_bytes\":71065600,\"used_in_bytes\":59825972,\"size_limit\":750780416},\"resident_set_size_in_bytes\":87064576},\"uptime_in_millis\":533268},\"requests\":{\"disconnects\":0,\"total\":1,\"status_codes\":{\"200\":1}},\"response_times\":{\"average\":17,\"max\":17},\"timestamp\":\"2017-02-03T15:29:08.500Z\"}\n","statusCode":403,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [update-monitor]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [update-monitor]\"},\"status\":403}"}
    at respond (E:\kibana-5.2.0-windows-x86\node_modules\elasticsearch\src\lib\transport.js:289:15)
    at checkRespForFailure (E:\kibana-5.2.0-windows-x86\node_modules\elasticsearch\src\lib\transport.js:248:7)
    at HttpConnector.<anonymous> (E:\kibana-5.2.0-windows-x86\node_modules\elasticsearch\src\lib\connectors\http.js:164:7)
    at IncomingMessage.wrapper (E:\kibana-5.2.0-windows-x86\node_modules\elasticsearch\node_modules\lodash\lodash.js:4968:19)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)

About my cluster:

I have a separate 1 node monitoring cluster and a 7 node main cluster. Each node has x-pack installed and I have a single client node with Kibana installed on it that's hooking into the monitoring cluster.

[Court (Court Ewing)]

The users and roles UI in Kibana only affects the cluster that stores the .kibana index, so it won't make modifications to separate monitoring clusters. You'll need to create a user with the monitoring_user role with the same username/password on your monitoring cluster. You can do that directly via the Elasticsearch API, or you can set up a separate Kibana instance to point to your monitoring cluster itself and configure the user through that.

So to summarize: if you're using a separate monitoring cluster, you'll need to have the same username/password on both clusters, and the user on the monitoring cluster will need to have the monitoring_user role.

[sredne]

Hi Court,

It is the same username for both clusters and has the same passwords and it's still giving me the same error.

EDIT:

The error still occurs in the backend but I can see the frontend monitoring now. I'm not sure why that is, but it looks to be partially resolved for now.

[tsullivan (Tim Sullivan)]

Hi sredne,

The error message you see in the Kibana log is regarding the feature of X-Pack Monitoring that allows the Kibana process to send it's operation statistics to the production cluster, to have them exported to the monitoring cluster. When this is correctly set up, you will see Kibana dashboards in the Monitoring UI.

The error tells me that there's a chance you changed the settings in kibana.yml for elasticsearch.username and elasticsearch.password. The recommended user for those settings is the built-in kibana user. You should change the password for that user so that you're not using the default, but you should still use that user for having the Kibana server connect to Elasticsearch.

Hope that helps,
-Tim