[ Creating new rule ]: ERROR Authentication using apikey failed - api key has been invalidated

[TheHunter1]

Hello,

I am using elasticsearch, kibana and beats version 7.10.1 with a Trial license,

I am trying to create a new Match rule and in my elasticsearch node I am getting this error:

[2021-01-18T11:33:18,032][WARN ][o.e.x.s.a.AuthenticationService] [DATA-03] Authentication using apikey failed - api key has been invalidated
[2021-01-18T11:33:18,035][WARN ][o.e.x.s.a.AuthenticationService] [DATA-03] Authentication using apikey failed - api key has been invalidated

and in kibana interface:

Rule failure at
Jan 18, 2021 @ 11:33:18.034

Reason: execute Message: [security_exception] missing authentication credentials for REST request [/_security/user/_has_privileges], with { header={ WWW-Authenticate={ 0="Basic realm=\"security\" charset=\"UTF-8\"" & 1="Bearer realm=\"security\"" & 2="ApiKey" } } }

My rule's configuration is like that:

Could you tell me please why I am getting this error ?
Thanks

[Frank_Hassanabad]

In the current 7.10.x series when a long running rule is still running and you go to update the rule while it is currently running, the API key which represents the rule is invalidated immediately which can result in that rule execution immediately showing an error within that current rule run.

The next rule run interval should clear out the error and you should see it run correctly. If on the second rule run it does not clear out and you are consistently seeing this error appear again and again (without editing the rule), then we might have something else we need to investigate deeper.

This rule is one of our first rules that can run for a very long time in which case when you modify it while it is currently running, this error shows up for 7.10.x series. But I want to make sure that if you wait for the next interval and the second rule run it does indeed clear out the error message as we expect.

The great news here is it looks to have been fixed in the soon to be released 7.11.0 with more advanced ways of deleting API keys:

Advanced API Key invalidation

Used SO for saving the API key IDs that should be deleted

If your error does not clear out for some odd reason on the next interval (which I think it really will), then you can re-create a new rule with the same settings from scratch and have that rule execute immediately and should not see this error message at all.

[TheHunter1]

Thanks for your answer @Frank_Hassanabad,

so I deleted the rule and created a new one from scrarch and I didn't see that error anymore, I have configured the rule to run every 5 minutes.

and I waited like 30 minutes, it has been executed and I see the results,

but It steal display Going to run and never display succeeded like the other rules

Should I just ignore this, or there is something wrong ?

Best regards.

[Frank_Hassanabad]

Going to run constantly being shown is really the rule itself saying, "I am running right now but am taking a long time to run". We have some open tickets for fixing the status of rules to be more clear about them being in the run state vs. in the "going to run" state as that's obviously confusing.

If it stays in the "going to run" state for a very long time then that means it is running for a very long period of time which isn't surprising since that does a many to many type of join based query which is going to take a while to run.

But it is normal and expected. If you want to break the rule apart using different queries such as querying for a smaller timeframe against your intelligence list that could potentially speed it up but can also have the opposite effect where you end up doing too many queries. But that's kind of up to how you want to tune it when it runs.

[TheHunter1]

Thanks for your answer @Frank_Hassanabad,

I Think that I am gonna try to break the rule into simple queries (without OR), as my cluster went down after 30min running that query, the CPU was at 100% in my 3 data nodes, and went all down