I'm having exactly the same issue. For what I've been able understand, the problem is not the user role or permission but instead the error in the search API request:
For instance, this API request method from the log:
/%3A.monitoring-es-6-%2C*%3A.monitoring-es-7-%2C.monitoring-es-6-%2C.monitoring-es-7-*/_search
is the problematic request. I crafted multiple requests based on the one above to find out where the problem is by removing part of it. Basically, the request above is doing a multiple index search be separating index patterns with , (%2C) and : (%3A)
In my particular case, this is the path:
/*%3A.monitoring-es-6-*%2C*%3A.monitoring-es-7-*%2C.monitoring-es-6-*%2C.monitoring-es-7-*/_search
removing url encoded string:
/*:.monitoring-es-6-*,*:.monitoring-es-7-*,.monitoring-es-6-*,.monitoring-es-7-*/_search
Resulting in an error
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
},
{
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
},
{
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
},
{
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
},
{
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
},
{
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : ".monitoring-es-7-2020.08.06",
"reason" : {
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
}
}
},
{
"shard" : 0,
"index" : ".monitoring-es-7-2020.08.07",
"reason" : {
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
}
}
},
{
"shard" : 0,
"index" : ".monitoring-es-7-2020.08.08",
"reason" : {
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
}
}
},
{
"shard" : 0,
"index" : ".monitoring-es-7-2020.08.09",
"reason" : {
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
}
}
},
{
"shard" : 0,
"index" : ".monitoring-es-7-2020.08.10",
"reason" : {
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
}
}
},
{
"shard" : 0,
"index" : ".monitoring-es-7-2020.08.11",
"reason" : {
"type" : "security_exception",
"reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
}
}
}
]
},
"status" : 403
}
If I manually remove : from the path request and try that again in dev tools it works:
POST /*.monitoring-es-6-*,.monitoring-es-7-*,.monitoring-es-6-*,.monitoring-es-7-*/_search
Anyways, that's the cause of the error but I'm not sure why it's happening, where is *: being appended and if it's a correct path syntax or not.
POST /*:/_search doesn't looks OK to me, but I honestly don't know.
@chrisronline Any clues?