Authentication using apikey failed

Elastic Stack Elasticsearch
5

In that case, you can try HttpTracer which will report the remote address of each request.

PUT _cluster/settings
{
   "persistent" : {
      "logger.org.elasticsearch.http.HttpTracer" : "TRACE"
   }
}

The logs will get quite verbose. You should see something like the follows for the failed API key authentication

received request from [Netty4HttpChannel{localAddress=/[0:0:0:0:0:0:0:1]:9200, remoteAddress=/[0:0:0:0:0:0:0:1]:53497}]org.elasticsearch.http.HttpHeadersValidationException: org.elasticsearch.ElasticsearchSecurityException: unable to authenticate with provided credentials and anonymous access is not allowed for this request

Hopefully the remote address will give you enough information to identify the agent that is using the invalid API key. Once you are done, you can remove HTTP tracer logging with

PUT _cluster/settings
{
   "persistent" : {
      "logger.org.elasticsearch.http.HttpTracer" : null
   }
}