Authentication using apikey failed

[dmrlixos]

Hi

I have a cluster with 3 instances ( 1 Master 2 Data Nodes )
Recenty looking into my cluster, i found a lot of warnings about Authentication using apikey failed on specific apikey id EjkscocB14********

I try search more about this apikey, but she not present on my api keys, probabily the reason of the error.

How i find who are using this api key ?

I using fleet with some integrations like windows,iis, nginx.

How resolve this warn error ?
Is possible do this ? Someone can help me ?

Thks

[dmrlixos]

someone can help, or have some idea to fix this?

[stephenb (Stephen Brown)]

Hi @dmrlixos

Apologies but there is not much to go on.

Can you share the entire authentication message?

You can turn up regular logging to to get more details ser here

Probably set

PUT /_cluster/settings
{
  "persistent": {
    "logger.org.elasticsearch.security": "DEBUG"
  }
}

Or perhaps

"logger.org.elasticsearch.security.authc": "DEBUG"

If you have a commercial subscription you turn in audit logging.

Yes this may have to do with fleet or an agent that was not updated/ properly removed etc... So the API Key is out of date, you can look on the source side.

You need to dig in and / or provide more detail

[dmrlixos]

Hi @stephenb

About the full message WARN.

May 23, 2023 2:10:46 AM WARN Server Authentication using apikey failed - unable to find apikey with id EjkscocB1498qJ03nQl***

Environment:
i'm using fleet with some integrations like windows,linux,iis to get theses logs by elastic agents.

I think too, that this message it's about some fleet agent that was be uninstalled incorrect, your explain make sense, but I don't have idea how agent is, I just have the api key id.

Subscription
I'm using a free and the audit logs is not allowed.

If want more details, please tell me

Thanks.

[Yang_Wang]

@dmrlixos

I'm using a free and the audit logs is not allowed.

In that case, you can try HttpTracer which will report the remote address of each request.

PUT _cluster/settings
{
   "persistent" : {
      "logger.org.elasticsearch.http.HttpTracer" : "TRACE"
   }
}

The logs will get quite verbose. You should see something like the follows for the failed API key authentication

received request from [Netty4HttpChannel{localAddress=/[0:0:0:0:0:0:0:1]:9200, remoteAddress=/[0:0:0:0:0:0:0:1]:53497}]org.elasticsearch.http.HttpHeadersValidationException: org.elasticsearch.ElasticsearchSecurityException: unable to authenticate with provided credentials and anonymous access is not allowed for this request

Hopefully the remote address will give you enough information to identify the agent that is using the invalid API key. Once you are done, you can remove HTTP tracer logging with

PUT _cluster/settings
{
   "persistent" : {
      "logger.org.elasticsearch.http.HttpTracer" : null
   }
}

[dmrlixos]

Hi @Yang_Wang

I enabled the trace http, but the error still are displayed like bellow

Authentication using apikey failed - unable to find apikey with id EjkscocB1498qJ03nQ***

The trace not show the full request, but from others requests are displayed fine, like this.

[null][OK][application/json][547] sent response to [Netty4HttpChannel{localAddress=/10...:9200, remoteAddress=/172..*.*1:18540}] success [true]

this