"reset" ILM failed step

Elastic Stack Elasticsearch
,
1

Hello,

I've some restricted indices stuck on ILM actions because of permissions.

security_exception: action [indices:admin/delete] is unauthorized for user [xxx] with roles [superuser] on restricted indices [.ds-.fleet-actions-results-2022.09.16-000001], this action is granted by the index privileges [delete_index,manage,all]

Some other displays set_priority.

I don't even know how the indices became on this state. Maybe some testing (and terribly failing!) on ILM for these indices.

I don't want to delete them, just solve this issue that is occuring everytime "ILM cron" runs (so everytime). I've tried to restore them from a previous snapshot but they are restricted indices so the same protection came in place and I'm unable to restore them.

Thank you, Pedro

2

Do the superuser role on your system include the mentioned index privileges?

3

Yes. This user is "Global / Super" admin.
The question is these are restricted indices as stated in the error output. They are supposed to not be manipulated by any kind of user despite their privileges. And I don't know how they have been caught (wrongly) on this ILM...

4

The .ds-.fleet-actions-results-2022.09.16-000001 index is a backing index of the .fleet-actions-results system data stream. It should only be managed by the elastic/fleet-server service account internally. I suggest you remove this index/datastream from the ILM policy that encountered this error.

5

We've already done that.
But how to clear the ILM stage error?

6

Do you know who created (or last modified) the ILM policy that is assigned to this '.ds-.fleet-actions-results-2022.09.16-000001' index? I have a suspicion that any actions that a policy specify is carried out with the privileges of whoever created/modified the policy.

As for clearing the error, have you tried opening up the index in the Kibana Index Management view, and clicked Manage Index -> Retry Policy (bottom right corner)? I don't know if it will do anything. Otherwise, check if the index has an alias (.ds-.fleet-actions-results)? If so, try deleting the ILM Policy from the index (same button), and then add it again.

7

Thanks for your feedback @jba.

I've tried to make "Remove policy" and the error message is similar than the error that the ILM Delete phase register in logs:

403: Forbidden. action [indices:admin/ilm/remove_policy] is unauthorized for user [XXXXXX] with effective roles [superuser] on restricted indices [.ds-.fleet-actions-results-000001], this action is granted by the index privileges [manage_ilm,manage,all]

So it seems that like this is a restricted indice, elastic doesn't allow to make any kind of action on it (Defining roles | Elasticsearch Guide [8.7] | Elastic).

I understand the reasons why we shouldn't delete them but keep this error going on and on and on is insane! The affected ILM is already correct and the failed step removed from it. Should have a way to clear the ILM failed step thought.

8

Final solution at last!

Have created a new (user) Role with "allow_restricted_indices": true permission and "all" on the restricted indice, like this:

POST /_security/role/_temp_restricted_indice_permission
{
  "indices": [
    {
      "names": [ ".restricted-indice-name" ],
      "privileges": [ "all" ],
      "allow_restricted_indices": true
    }
  ]
}

Then go to Kibana -> Management -> Users and give this role to your user.

Next go to Kibana -> Management -> Index Management, filter by indices with errors, go to detail to each one of the indices and click "Manage ^" -> "Remove lifecycle policy" like shown bellow:


imagem


Solve this issue, no more indices with errors and no restricted indices or system configuration / information was affected! :slight_smile:

Thank you all for your inputs and shared ideas.

Greetings, Pedro


P.S. - maybe just manage_ilm or manage permissions on (user) Role catching the restricted indice will do the job but...

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.