Hi,
Recently I couldn't open Stack Monitoring. Then I decided to upgrade to new version 7.8.0. But after upgrading my problem still exists.
My user has role superuser so I don't understand why I cannot have access to monitoring indices.
Can you share the roles your user has?
The only one role superuser.
Chris Roberson via Discuss the Elastic Stack elastic@discoursemail.com 23 июня 2020 г. 17:05:04 написал:
That's strange. If you are using the elastic superuser account, there should never be permission issues.
Are you using a dedicated, separate monitoring cluster? Can you share your kibana.yml?
We use one cluster with all data and monitoring indices placed together.
My kibana.yml
server.host: "0.0.0.0"
elasticsearch.hosts:
kibana.index: ".kibana"
elasticsearch.username: "kibana_user"
elasticsearch.password: "**************"logging.dest: /var/log/kibana/kibana.log
xpack.monitoring.enabled: true
xpack.monitoring.ui.enabled: true
Try creating a user with the roles defined in the error message (kibana_admin and monitoring_user) then logging in as that user. Does that fix it?
I created a new user and added 2 roles, but have the same error.
Hmm. Can you double check there aren't any errors in either the Kibana or Elasticsearch server log that might help explain this?
Well, elastic logs are clear but while I try to open stack monitoring in kibana I see many errors in kibana log
{"type":"log","@timestamp":"2020-07-15T16:43:32Z","tags":["error","plugins","monitoring","monitoring"],"pid":9192,"message":"{ Error: [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [user] (and) [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [user] (and) [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [user] (and) [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [user] (and) [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [user] (and) [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [user] (and) [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [user]\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)\n status: 403,\n displayName: 'AuthorizationException',\n message:\n '[security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [user] (and) [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [user] (and) [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [user] (and) [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [user] (and) [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [user] (and) [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [user] (and) [security_exception] action [indices:data/read/search[can_match]] is unauthorized for user [user]',\n path:\n '/%3A.monitoring-es-6-%2C*%3A.monitoring-es-7-%2C.monitoring-es-6-%2C.monitoring-es-7-*/_search',\n query:\n { size: 10000,\n ignore_unavailable: true,\n filter_path:\n 'hits.hits._index,hits.hits._source.cluster_uuid,hits.hits._source.cluster_name,hits.hits._source.version,hits.hits._source.license.status,hits.hits._source.license.type,hits.hits._source.license.issue_date,hits.hits._source.license.expiry_date,hits.hits._source.license.expiry_date_in_millis,hits.hits._source.cluster_stats,hits.hits._source.cluster_state,hits.hits._source.cluster_settings.cluster.metadata.display_name' },\n body:\n { error:\n { root_cause: [Array],\n type: 'search_phase_execution_exception',\n reason: 'all shards failed',\n phase: 'can_match',\n grouped: true,\n failed_shards: [Array] },\n status: 403 },\n statusCode: 403,\n response:\n '{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]"},{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"can_match","grouped":true,"failed_shards":[{"shard":0,"index":".monitoring-es-7-2020.07.09","node":"Zy4jPFsyR_S6mvlUQJOxDg","reason":{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]","caused_by":{"type":"illegal_state_exception","reason":"There are no external requests known to support wildcards that don\'t support replacing their indices"}}},{"shard":0,"index":".monitoring-es-7-2020.07.10","node":"0HC5iE8ITcqRY7kFPem6rQ","reason":{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]","caused_by":{"type":"illegal_state_exception","reason":"There are no external requests known to support wildcards that don\'t support replacing their indices"}}},{"shard":0,"index":".monitoring-es-7-2020.07.11","node":"0HC5iE8ITcqRY7kFPem6rQ","reason":{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]","caused_by":{"type":"illegal_state_exception","reason":"There are no external requests known to support wildcards that don\'t support replacing their indices"}}},{"shard":0,"index":".monitoring-es-7-2020.07.12","node":"9FcVASLhQaecOi41I6Nu0g","reason":{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]","caused_by":{"type":"illegal_state_exception","reason":"There are no external requests known to support wildcards that don\'t support replacing their indices"}}},{"shard":0,"index":".monitoring-es-7-2020.07.13","node":"Zy4jPFsyR_S6mvlUQJOxDg","reason":{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]","caused_by":{"type":"illegal_state_exception","reason":"There are no external requests known to support wildcards that don\'t support replacing their indices"}}},{"shard":0,"index":".monitoring-es-7-2020.07.14","node":"0HC5iE8ITcqRY7kFPem6rQ","reason":{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]","caused_by":{"type":"illegal_state_exception","reason":"There are no external requests known to support wildcards that don\'t support replacing their indices"}}},{"shard":0,"index":".monitoring-es-7-2020.07.15","node":"0HC5iE8ITcqRY7kFPem6rQ","reason":{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]","caused_by":{"type":"illegal_state_exception","reason":"There are no external requests known to support wildcards that don\'t support replacing their indices"}}}]},"status":403}',\n toString: [Function],\n toJSON: [Function] }"}
{"type":"response","@timestamp":"2020-07-15T16:43:32Z","tags":,"pid":9192,"method":"post","statusCode":403,"req":{"url":"/api/monitoring/v1/clusters","method":"post","headers":{"connection":"upgrade","host":"kibana.domain.local","content-length":"101","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0","accept":"application/json, text/plain, /","accept-language":"ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate, br","content-type":"application/json;charset=utf-8","kbn-version":"7.8.0","origin":"https://kibana.domain.local","referer":"https://kibana.domain.local/app/monitoring"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://kibana.domain.local/app/monitoring"},"res":{"statusCode":403,"responseTime":76,"contentLength":9},"message":"POST /api/monitoring/v1/clusters 403 76ms - 9.0B"}
{"type":"response","@timestamp":"2020-07-15T16:43:32Z","tags":,"pid":9192,"method":"get","statusCode":200,"req":{"url":"/api/monitoring/v1/check_access","method":"get","headers":{"connection":"upgrade","host":"kibana.domain.local","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0","accept":"application/json, text/plain, /","accept-language":"ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3","accept-encoding":"gzip, deflate, br","kbn-version":"7.8.0","referer":"https://kibana.domain.local/app/monitoring"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://kibana.domain.local/app/monitoring"},"res":{"statusCode":200,"responseTime":59,"contentLength":9},"message":"GET /api/monitoring/v1/check_access 200 59ms - 9.0B"}
I replaced real host name and user login.
Hmm okay. So for the user you created, can you please run these commands and return all the results?
GET _security/user/{THE_USER_YOU_CREATED}
Use the roles list from the above response to run the next command:
GET _security/role/{THE_ROLE_FROM_ABOVE_AS_COMMA_SEPARATED_LIST}
Maybe my problem is wider.
I tried to open monitoring many times and opened. I see
What concerns your question:
{
"user" : {
"username" : "user",
"roles" : [
"superuser",
"wazuh_admin"
],
"full_name" : "User",
"email" : "",
"metadata" : { },
"enabled" : true
}
}
{
"superuser" : {
"cluster" : [
"all"
],
"indices" : [
{
"names" : [
"*"
],
"privileges" : [
"all"
],
"allow_restricted_indices" : true
}
],
"applications" : [
{
"application" : "*",
"privileges" : [
"*"
],
"resources" : [
"*"
]
}
],
"run_as" : [
"*"
],
"metadata" : {
"_reserved" : true
},
"transient_metadata" : { }
}
}
{
"wazuh_admin" : {
"cluster" : [ ],
"indices" : [
{
"names" : [
"wazuh-*"
],
"privileges" : [
"all"
],
"field_security" : {
"grant" : [
"*"
],
"except" : [ ]
},
"allow_restricted_indices" : false
}
],
"applications" : [ ],
"run_as" : [ ],
"metadata" : { },
"transient_metadata" : {
"enabled" : true
}
}
}Maybe this info helps
"action [indices:data/read/search[can_match]] is unauthorized for user [user]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"can_match","grouped":true,"failed_shards":[{"shard":0,"index":".monitoring-es-7-2020.07.09","node":"0HC5iE8ITcqRY7kFPem6rQ","reason":{"type":"security_exception","reason":"action [indices:data/read/search[can_match]] is unauthorized for user [user]","caused_by":{"type":"illegal_state_exception","reason":"There are no external requests known to support wildcards that don\'t support replacing their indices"}}}
All monitoring indices are accessible
I honestly don't know.
Can you try recreating your environment to see if the error still happens? I'm going to tag the Elasticsearch team on this as well as they might be able to help more.
It's a cluster from 6 servers so it's not simple to recreate it.
I have another one standalone server with Elasticsearch stack the same version and this error is not reproduced there.
I'm having exactly the same issue. For what I've been able understand, the problem is not the user role or permission but instead the error in the search API request:
For instance, this API request method from the log:
/%3A.monitoring-es-6-%2C*%3A.monitoring-es-7-%2C.monitoring-es-6-%2C.monitoring-es-7-*/_search
is the problematic request. I crafted multiple requests based on the one above to find out where the problem is by removing part of it. Basically, the request above is doing a multiple index search be separating index patterns with , (%2C) and : (%3A)
In my particular case, this is the path:
/*%3A.monitoring-es-6-*%2C*%3A.monitoring-es-7-*%2C.monitoring-es-6-*%2C.monitoring-es-7-*/_search
removing url encoded string:
/*:.monitoring-es-6-*,*:.monitoring-es-7-*,.monitoring-es-6-*,.monitoring-es-7-*/_search
Resulting in an error
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
},
{
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
},
{
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
},
{
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
},
{
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
},
{
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : ".monitoring-es-7-2020.08.06",
"reason" : {
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
}
}
},
{
"shard" : 0,
"index" : ".monitoring-es-7-2020.08.07",
"reason" : {
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
}
}
},
{
"shard" : 0,
"index" : ".monitoring-es-7-2020.08.08",
"reason" : {
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
}
}
},
{
"shard" : 0,
"index" : ".monitoring-es-7-2020.08.09",
"reason" : {
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
}
}
},
{
"shard" : 0,
"index" : ".monitoring-es-7-2020.08.10",
"reason" : {
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
}
}
},
{
"shard" : 0,
"index" : ".monitoring-es-7-2020.08.11",
"reason" : {
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
}
}
}
]
},
"status" : 403
}
If I manually remove : from the path request and try that again in dev tools it works:
POST /*.monitoring-es-6-*,.monitoring-es-7-*,.monitoring-es-6-*,.monitoring-es-7-*/_search
Anyways, that's the cause of the error but I'm not sure why it's happening, where is *: being appended and if it's a correct path syntax or not.
POST /*:/_search doesn't looks OK to me, but I honestly don't know.
@chrisronline Any clues?
Further debugging:
Looks that:
POST /*:.monitoring*
works OK, but this fails:
POST /*:.monitoring-es-7-*,.monitoring-es-7-*/_search
And both individually works ok:
POST /*:.monitoring-es-7-*/_search
POST /.monitoring-es-7-*/_search
I'm really puzzled about this.
If you are seeing this message, then it's definitely a bug.
Are you able to capture a HAR from your browser when this error occurs?
If it has sensitive information in it (it probably will), then you can send it to me via a private message.
Or, if you are a paid customer you can open a ticket on our support portal and provide the details there.
hi @TimV
just to clarify, is this request valid?
POST /*:.monitoring-es-7-*,.monitoring-es-7-*/_search
Because that looks like the request is being made by kibana.
If you split that request in two, i.e.:
POST /*:.monitoring-es-7-*/_search
POST /.monitoring-es-7-*/_search
it looks fine.
I sent you a private message with link to HAR.
Hi!
Any news on this issue?