Hi guys!
New user here, I'm currently finding a tonne of this in my elasticsearch_server.json log, and I can't find out where it is:
{"type": "server", "timestamp": "2021-03-03T11:10:08,518+01:00", "level": "WARN", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elasticsearch", "node.name": "node-1", "message": "Authentication using apikey failed - invalid ApiKey value", "cluster.uuid": "UbN_iEEITBuzgDzs
-em7Ww", "node.id": "U-ZizcsnT4yBtKHF3xomwQ" ,
"stacktrace": ["java.lang.IllegalArgumentException: invalid ApiKey value",
"at org.elasticsearch.xpack.security.authc.ApiKeyService.getCredentialsFromHeader(ApiKeyService.java:701) ~[x-pack-security-7.10.0.jar:7.10.0]",
"at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:355) [x-pack-security-7.10.0.jar:7.10.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:347) [x-pack-security-7.10.0.jar:7.10.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:329) [x-pack-security-7.10.0.jar:7.10.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:405) [x-pack-security-7.10.0.jar:7.10.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:325) [x-pack-security-7.10.0.jar:7.10.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) [x-pack-security-7.10.0.jar:7.10.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) [x-pack-security-7.10.0.jar:7.10.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) [x-pack-security-7.10.0.jar:7.10.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) [x-pack-security-7.10.0.jar:7.10.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:141) [x-pack-security-7.10.0.jar:7.10.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:126) [x-pack-security-7.10.0.jar:7.10.0]",
"at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:70) [x-pack-security-7.10.0.jar:7.10.0]",
"at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:258) [elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:340) [elasticsearch-7.10.0.jar:7.10.0]",
"at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:191) [elasticsearch-7.10.0.jar:7.10.0]",
I'm trying to figure out which host this comes from, is it really the elastichost itself? And why? I have xpack enabled, and we are running kibana/filebeat and it seems to work, so I'm not sure which host could be the culprit, so I want to know how to find it out.
Here is the elastic config:
> node.name: node-1
> path.data: /var/lib/elasticsearch
> path.logs: /var/log/elasticsearch
> cluster.initial_master_nodes: node-1
> network.host: 0.0.0.0
> xpack.security.enabled: true
> xpack.security.authc.api_key.enabled: true
> xpack.security.http.ssl.enabled: true
> xpack.security.transport.ssl.enabled: true
> xpack.security.http.ssl.key: /etc/elasticsearch/certs/foo.key
> xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/foo.crt
> xpack.security.transport.ssl.key: /etc/elasticsearch/certs/foo.key
> xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/foo.crt