ILM `security_exception` error

[jsu]

Hey there,

So, I currently have a cluster running for timeseries with security enabled.

There's one main ILM policy, let's call it policy1:

"policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "0ms",
          "actions" : {
            "allocate" : {
              "number_of_replicas" : 1,
              "include" : { },
              "exclude" : { },
              "require" : {
                "temp" : "hot"
              }
            },
            "forcemerge" : {
              "max_num_segments" : 1
            },
            "set_priority" : {
              "priority" : 100
            },
            "shrink" : {
              "number_of_shards" : 1
            }
          }
        },
        "cold" : {
          "min_age" : "23d",
          "actions" : {
            "allocate" : {
              "number_of_replicas" : 0,
              "include" : { },
              "exclude" : { },
              "require" : {
                "temp" : "warm"
              }
            },
            "freeze" : { },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "7d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "83d",
          "actions" : {
            "delete" : { }
          }
        }
      }
    }

I use 2 different users to work on this cluster:

• user1 which has a lot of privileges, but no superuser privileges.
• superuser which has...well, superuser privileges.

I guess I tried to do something which I couldn't with user1, because now I have a lot of ILM security_exception errors:

 "reason" : "action [indices:admin/settings/update] is unauthorized for user [user1]"

To fix this, I tried the following:

• log in with user1 and retry ILM step.
• log in with superuser and retry ILM step
• log in with superuser, give user1 superuser privileges, log out, log back in with user1 and retry ILM step
• log in with superuser and use the move_to_step API (after finding what to put under "name")

And nothing.

Then I tried to create a new ILM policy, policy2, copying the policy1 but without rollover, so without hot phase. I removed policy1 from a problematic index and applied the new policy2. So far, so good, I don't have any error, but I'm not sure from when the "min_age" will be counted, so I'm not sure if it's normal that my index is still not deleted or not.

The last weird thing I noticed is that I created a whole new index template and bootstrapped a new index with the policy1 applied to it, all this while being logged in with superuser.
A few minutes later, it already appears that it has the same ILM error on it (on the set_priority step):

"reason" : "action [indices:admin/settings/update] is unauthorized for user [user1]"

This I really don't understand as user1 was never involved in anything about this index.

Does anyone as an idea of what is happening here?

Thanks in advance for your help!

Cheers,

jsu

[dakrone (Lee Hinman)]

When the ILM policy is created, ES stores the credentials that were used by whatever user created the policy. These credentials are what is used for executing ILM actions.

It sounds like with your current users you can do PUT /_ilm/policy1 with your superuser user (it can even be the same policy body), which will then have permissions to perform any of the ILM actions.

[jsu]

Thanks a lot @dakrone !

Indeed, I thought that updating my user1 privileges would solve this as the original creator of the policy now had enough privileges for ILM actions. This didn't work.

But indeed updating the policy (exact same body, as you suggested) was the way to go! I could then "retry lifecycle step" for every problematic index (doing it in bulk didn't work for some reason) and everything worked like a charm. Thanks again!