ECK apm server failed to to index event (security_exception): action [indices:admin/auto_create]

Elastic Observability APM
, ,
1

Kibana version:8.5.3

Elasticsearch version:8.5.3

APM Server version:8.5.3
I used the below code for ECK APM
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
name: apm-server-stg
spec:
version: 8.5.3
count: 1
elasticsearchRef:
name: elasticsearch
kibanaRef:
name: kibana-stg
config:
http:
enabled: true

I am getting the below error logs from APM server

'''{"log.level":"error","@timestamp":"2023-09-25T09:47:27.142Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":412},"message":"failed to index event (security_exception): action [indices:admin/auto_create] is unauthorized for user [apm-server-apm-user] with effective roles [apm_system,eck_apm_user_role_v75,ingest_admin] on indices [metrics-apm.app.xxxx, this action is granted by the index privileges [auto_configure,create_index,manage,all]","service.name":"apm-server","ecs.version":"1.6.0"}.'''

My issue is similar to this topic APM Server failed to index event (security_exception): action [indices:admin/auto_create] is unauthorized for user
Please someone help me on this issue.

2

@jijil It looks like the user apm-server-apm-user, configured to access Elasticsearch on behalf of APM-Server doesn't have the correct privileges. Is the APM-Server manifest you shared the full manifest? Also, can you share the full APM-Server logs (removing any confidential information) and the ECK version you are using?

3

Hi @lahsivjar , I used the above apm-server manifest to deploy the apm server. I am using eck version of 8.5.3. Can i need to add anything in my manifest for accessing Elasticsearch on behalf of APM-Server? Added apm agent to Java microservices running on AKS and it's able to send logs to apm server but APM is not able to send logs to Elasticsearch. Below is the full log of APM-server

{"log.level":"info","@timestamp":"2023-09-26T03:07:09.395Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"yyyyyyyy","http.request.id":"","event.duration":10012317358,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:09.966Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"yyyyyyyy","http.request.id":"","event.duration":10021940797,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:10.557Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"yyyyyyyy","http.request.id":"","event.duration":10009381529,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:11.665Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10011616046,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2023-09-26T03:07:12.670Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":412},"message":"failed to index event (security_exception): action [indices:admin/auto_create] is unauthorized for user [lms-apm-server-stg-apm-user] with effective roles [apm_system,eck_apm_user_role_v75,ingest_admin] on indices [metrics-apm.app.xxxxxx], this action is granted by the index privileges [auto_configure,create_index,manage,all]","service.name":"apm-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:12.728Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10025578421,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:13.591Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (xxxxxxx","source.address":"","http.request.id":"","event.duration":10005980859,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:14.116Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10010025957,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:14.324Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.25","source.address":"10.244.1.1","http.request.body.bytes":0,"http.request.id":"","event.duration":174502,"http.response.status_code":200,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:14.754Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"10.244.0.118","http.request.id":"","event.duration":10009038765,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2023-09-26T03:07:15.124Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":412},"message":"failed to index event (security_exception): action [indices:admin/auto_create] is unauthorized for user [lms-apm-server-stg-apm-user] with effective roles [apm_system,eck_apm_user_role_v75,ingest_admin] on indices [metrics-apm.app.xxxxx], this action is granted by the index privileges [auto_configure,create_index,manage,all]","service.name":"apm-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:15.714Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10010917864,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:16.143Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10019070266,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:17.107Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10025579442,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:17.568Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10006204501,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:18.562Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10014704461,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:18.829Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (pxxxxxxx)","source.address":"","http.request.id":"","event.duration":10010175389,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2023-09-26T03:07:19.588Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":412},"message":"failed to index event (security_exception): action [indices:admin/auto_create] is unauthorized for user [lms-apm-server-stg-apm-user] with effective roles [apm_system,eck_apm_user_role_v75,ingest_admin] on indices [metrics-apm.app.xxxxx], this action is granted by the index privileges [auto_configure,create_index,manage,all]","service.name":"apm-server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2023-09-26T03:07:21.595Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10011097076,"http.response.status_code":202,"ecs.version":"1.6.0"}

4

Thanks for the reply.

I am using eck version of 8.5.3

I mean the ECK version, not the Elastic stack version. The latest is 2.9.0 and I would recommend you to use that if you are not already.

Can i need to add anything in my manifest for accessing Elasticsearch on behalf of APM-Server?

If Elasticsearch is also managed by ECK (which I am assuming is true), then, adding elasticsearchRef should be enough to correctly configure the credentials for APM-Server to access Elasticsearch. However, it looks like the privileges are not added correctly for your case.

is unauthorized for user [lms-apm-server-stg-apm-user] with effective roles [apm_system,eck_apm_user_role_v75,ingest_admin] on indices [metrics-apm.app.xxxxxx], this action is granted by the index privileges [auto_configure,create_index,manage,all]

This line from the logs is especially concerning. The log states that the user lms-apm-server-stg-apm-user has the role eck_apm_user_role_v75, however, since you are on 8.x it should be using eck_apm_user_role_v80 (or something similar). Did you upgrade the cluster from 7.x to 8.x recently?

IIRC, in ECK, Elasticsearch uses file-based authentication. It might be possible that the files are not correctly loaded into Elasticsearch causing issues with the roles assigned to to user.

As a next step, we should look into these files and figure out where the issue is (are the files not updated or has ES failed to load the files). To do this, we can either SSH into the ES nodes and look at the mounted files (mostly at /usr/share/elasticsearch/config/) OR find a Kubernetes secret named *-file-realm and decode that. Our expectation is that the user lms-apm-server-stg-apm-user should have the role like eck_apm_user_role_v80 (rather than the one ending in v75).

5

My ECK version is 1.7.1 . I didn't upgrade the cluster recently .
From the elasticsearch-es-xpack-file-realm secret, I can see the below role eck_apm_user_role_v75

eck_apm_agent_user_role:
  cluster: []
  indices: []
  applications:
  - application: kibana-.kibana
    privileges:
    - feature_apm.read
    resources:
    - space:default
eck_apm_user_role_v6:
  cluster:
  - monitor
  - manage_index_templates
  indices:
  - names:
    - apm-*
    privileges:
    - write
    - create_index
  applications: []
eck_apm_user_role_v7:
  cluster:
  - monitor
  - manage_ilm
  - manage_index_templates
  indices:
  - names:
    - apm-*
    privileges:
    - manage
    - write
    - create_index
  applications: []
eck_apm_user_role_v75:
  cluster:
  - monitor
  - manage_ilm
  - manage_api_key
  indices:
  - names:
    - apm-*
    privileges:
    - manage
    - create_doc
    - create_index
  applications: []
6

I can see the below role eck_apm_user_role_v75

Hmm, it seems to be missing the eck_apm_user_role_v80 and that might be the source of the issues (but I am not sure about this). I am not an ECK expert so we can wait for someone more knowledgeable to chime in, however, based on my understanding, it looks like eck_apm_user_role_v80 was added in v2.2.0 of ECK release. The support matrix does show that we support 7.1+ for v1.7.x of ECK but not sure if that includes 8.0 too (ref).

If possible, can you try to upgrade to a newer ECK release? I think, another option would be to manually create a user with the correct privileges to access Elasticsearch for APM and configure that in the ApmServer manifest. Example:

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm-server-stg
spec:
  version: 8.5.3
  count: 1
  config:
    output:
      elasticsearch:
        username: admin
        password: changeme
  elasticsearchRef:
    name: elasticsearch
7

I have confirmed this point, 8.x versions for the Elastic stack are supported for 2.2+ ECK versions.

This leaves us with the 2 options I mentioned in the previous comment, let me know if you can use one of those.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.