# ECK apm server failed to to index event (security\_exception): action \[indices:admin/auto\_create\]
**URL:** https://discuss.elastic.co/t/eck-apm-server-failed-to-to-index-event-security-exception-action-indices-admin-auto-create/343741
**Category:** APM
**Tags:** elastic-stack-security, java, server
**Created:** [September 25, 2023, 10:09am UTC](https://discuss.elastic.co/t/eck-apm-server-failed-to-to-index-event-security-exception-action-indices-admin-auto-create/343741 "2023-09-25T10:09:38Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@jijil](https://discuss.elastic.co/u/jijil)
#### Post date: [September 25, 2023, 10:09am UTC](https://discuss.elastic.co/t/eck-apm-server-failed-to-to-index-event-security-exception-action-indices-admin-auto-create/343741/1 "2023-09-25T10:09:38Z")
**Kibana version** :8.5.3
**Elasticsearch version** :8.5.3
**APM Server version** :8.5.3
I used the below code for ECK APM
apiVersion: [apm.k8s.elastic.co/v1](http://apm.k8s.elastic.co/v1)
kind: ApmServer
metadata:
name: apm-server-stg
spec:
version: 8.5.3
count: 1
elasticsearchRef:
name: elasticsearch
kibanaRef:
name: kibana-stg
config:
http:
enabled: true
I am getting the below error logs from APM server
'''{"log.level":"error","@timestamp":"2023-09-25T09:47:27.142Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":412},"message":"failed to index event (security\_exception): action [indices:admin/auto\_create] is unauthorized for user [apm-server-apm-user] with effective roles [apm\_system,eck\_apm\_user\_role\_v75,ingest\_admin] on indices [metrics-apm.app.xxxx, this action is granted by the index privileges [auto\_configure,create\_index,manage,all]","service.name":"apm-server","ecs.version":"1.6.0"}.'''
My issue is similar to this topic [APM Server failed to index event (security\_exception): action [indices:admin/auto\_create] is unauthorized for user](https://discuss.elastic.co/t/apm-server-failed-to-index-event-security-exception-action-indices-admin-auto-create-is-unauthorized-for-user/301322)
Please someone help me on this issue.
---
### Author:  [@lahsivjar](https://discuss.elastic.co/u/lahsivjar)
#### Post date: [September 26, 2023, 2:56am UTC](https://discuss.elastic.co/t/eck-apm-server-failed-to-to-index-event-security-exception-action-indices-admin-auto-create/343741/2 "2023-09-26T02:56:34Z")
@jijil It looks like the user `apm-server-apm-user`, configured to access Elasticsearch on behalf of APM-Server doesn't have the correct privileges. Is the APM-Server manifest you shared the full manifest? Also, can you share the full APM-Server logs (removing any confidential information) and the ECK version you are using?
---
### Author:  [@jijil](https://discuss.elastic.co/u/jijil)
#### Post date: [September 26, 2023, 5:30am UTC](https://discuss.elastic.co/t/eck-apm-server-failed-to-to-index-event-security-exception-action-indices-admin-auto-create/343741/3 "2023-09-26T05:30:20Z")
Hi @lahsivjar , I used the above apm-server manifest to deploy the apm server. I am using eck version of 8.5.3. Can i need to add anything in my manifest for accessing Elasticsearch on behalf of APM-Server? Added apm agent to Java microservices running on AKS and it's able to send logs to apm server but APM is not able to send logs to Elasticsearch. Below is the full log of APM-server
{"log.level":"info","@timestamp":"2023-09-26T03:07:09.395Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"yyyyyyyy","http.request.id":"","event.duration":10012317358,"http.response.status\_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:09.966Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"yyyyyyyy","http.request.id":"","event.duration":10021940797,"http.response.status\_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:10.557Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"yyyyyyyy","http.request.id":"","event.duration":10009381529,"http.response.status\_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:11.665Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10011616046,"http.response.status\_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-26T03:07:12.670Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":412},"message":"failed to index event (security\_exception): action [indices:admin/auto\_create] is unauthorized for user [lms-apm-server-stg-apm-user] with effective roles [apm\_system,eck\_apm\_user\_role\_v75,ingest\_admin] on indices [metrics-apm.app.xxxxxx], this action is granted by the index privileges [auto\_configure,create\_index,manage,all]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:12.728Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10025578421,"http.response.status\_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:13.591Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (xxxxxxx","source.address":"","http.request.id":"","event.duration":10005980859,"http.response.status\_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:14.116Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10010025957,"http.response.status\_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:14.324Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user\_agent.original":"kube-probe/1.25","source.address":"10.244.1.1","http.request.body.bytes":0,"http.request.id":"","event.duration":174502,"http.response.status\_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:14.754Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"10.244.0.118","http.request.id":"","event.duration":10009038765,"http.response.status\_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-26T03:07:15.124Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":412},"message":"failed to index event (security\_exception): action [indices:admin/auto\_create] is unauthorized for user [lms-apm-server-stg-apm-user] with effective roles [apm\_system,eck\_apm\_user\_role\_v75,ingest\_admin] on indices [metrics-apm.app.xxxxx], this action is granted by the index privileges [auto\_configure,create\_index,manage,all]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:15.714Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10010917864,"http.response.status\_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:16.143Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10019070266,"http.response.status\_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:17.107Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10025579442,"http.response.status\_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:17.568Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10006204501,"http.response.status\_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:18.562Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10014704461,"http.response.status\_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:18.829Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (pxxxxxxx)","source.address":"","http.request.id":"","event.duration":10010175389,"http.response.status\_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-26T03:07:19.588Z","log.logger":"modelindexer","log.origin":{"file.name":"modelindexer/indexer.go","file.line":412},"message":"failed to index event (security\_exception): action [indices:admin/auto\_create] is unauthorized for user [lms-apm-server-stg-apm-user] with effective roles [apm\_system,eck\_apm\_user\_role\_v75,ingest\_admin] on indices [metrics-apm.app.xxxxx], this action is granted by the index privileges [auto\_configure,create\_index,manage,all]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-26T03:07:21.595Z","log.logger":"request","log.origin":{"file.name":"middleware/log\_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user\_agent.original":"apm-agent-java/1.42.0 (xxxxxxx)","source.address":"","http.request.id":"","event.duration":10011097076,"http.response.status\_code":202,"ecs.version":"1.6.0"}
---
### Author:  [@lahsivjar](https://discuss.elastic.co/u/lahsivjar)
#### Post date: [September 26, 2023, 6:57am UTC](https://discuss.elastic.co/t/eck-apm-server-failed-to-to-index-event-security-exception-action-indices-admin-auto-create/343741/4 "2023-09-26T06:57:04Z")
Thanks for the reply.
> I am using eck version of 8.5.3
I mean the [ECK version](https://www.elastic.co/guide/en/cloud-on-k8s/current/eck-release-notes.html), not the Elastic stack version. The latest is `2.9.0` and I would recommend you to use that if you are not already.
> Can i need to add anything in my manifest for accessing Elasticsearch on behalf of APM-Server?
If Elasticsearch is also managed by ECK (which I am assuming is true), then, adding `elasticsearchRef` should be enough to correctly configure the credentials for APM-Server to access Elasticsearch. However, it looks like the privileges are not added correctly for your case.
> is unauthorized for user [lms-apm-server-stg-apm-user] with effective roles [apm\_system,eck\_apm\_user\_role\_v75,ingest\_admin] on indices [metrics-apm.app.xxxxxx], this action is granted by the index privileges [auto\_configure,create\_index,manage,all]
This line from the logs is especially concerning. The log states that the user `lms-apm-server-stg-apm-user` has the role `eck_apm_user_role_v75`, however, since you are on `8.x` it should be using `eck_apm_user_role_v80` (or something similar). Did you upgrade the cluster from `7.x` to `8.x` recently?
IIRC, in ECK, Elasticsearch uses [file-based authentication](https://www.elastic.co/guide/en/elasticsearch/reference/current/file-realm.html). It might be possible that the files are not correctly loaded into Elasticsearch causing issues with the roles assigned to to user.
As a next step, we should look into these files and figure out where the issue is (are the files not updated or has ES failed to load the files). To do this, we can either SSH into the ES nodes and look at the mounted files (mostly at `/usr/share/elasticsearch/config/`) OR find a Kubernetes secret named `*-file-realm` and decode that. Our expectation is that the user `lms-apm-server-stg-apm-user` should have the role like `eck_apm_user_role_v80` (rather than the one ending in `v75`).
---
### Author:  [@jijil](https://discuss.elastic.co/u/jijil)
#### Post date: [September 26, 2023, 7:28am UTC](https://discuss.elastic.co/t/eck-apm-server-failed-to-to-index-event-security-exception-action-indices-admin-auto-create/343741/5 "2023-09-26T07:28:05Z")
> [@lahsivjar](#):
>
> ECK version
My ECK version is 1.7.1 . I didn't upgrade the cluster recently .
From the elasticsearch-es-xpack-file-realm secret, I can see the below role eck\_apm\_user\_role\_v75
```auto
eck_apm_agent_user_role:
cluster: []
indices: []
applications:
- application: kibana-.kibana
privileges:
- feature_apm.read
resources:
- space:default
eck_apm_user_role_v6:
cluster:
- monitor
- manage_index_templates
indices:
- names:
- apm-*
privileges:
- write
- create_index
applications: []
eck_apm_user_role_v7:
cluster:
- monitor
- manage_ilm
- manage_index_templates
indices:
- names:
- apm-*
privileges:
- manage
- write
- create_index
applications: []
eck_apm_user_role_v75:
cluster:
- monitor
- manage_ilm
- manage_api_key
indices:
- names:
- apm-*
privileges:
- manage
- create_doc
- create_index
applications: []
```
---
### Author:  [@lahsivjar](https://discuss.elastic.co/u/lahsivjar)
#### Post date: [September 26, 2023, 8:20am UTC](https://discuss.elastic.co/t/eck-apm-server-failed-to-to-index-event-security-exception-action-indices-admin-auto-create/343741/6 "2023-09-26T08:20:08Z")
> I can see the below role eck\_apm\_user\_role\_v75
Hmm, it seems to be missing the `eck_apm_user_role_v80` and that might be the source of the issues (but I am not sure about this). I am not an ECK expert so we can wait for someone more knowledgeable to chime in, however, based on my understanding, it looks like `eck_apm_user_role_v80` was added in `v2.2.0` of ECK release. The support matrix does show that we support `7.1+` for `v1.7.x` of ECK but not sure if that includes `8.0` too ([ref](https://www.elastic.co/guide/en/cloud-on-k8s/1.7/k8s_supported_versions.html)).
If possible, can you try to upgrade to a newer ECK release? I think, another option would be to manually create a user with the correct privileges to access Elasticsearch for APM and configure that in the ApmServer manifest. Example:
```yaml
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
name: apm-server-stg
spec:
version: 8.5.3
count: 1
config:
output:
elasticsearch:
username: admin
password: changeme
elasticsearchRef:
name: elasticsearch
```
---
### Author:  [@lahsivjar](https://discuss.elastic.co/u/lahsivjar)
#### Post date: [September 26, 2023, 9:19am UTC](https://discuss.elastic.co/t/eck-apm-server-failed-to-to-index-event-security-exception-action-indices-admin-auto-create/343741/7 "2023-09-26T09:19:41Z")
> [@lahsivjar](#):
>
> The support matrix does show that we support `7.1+` for `v1.7.x` of ECK but not sure if that includes `8.0` too ([ref](https://www.elastic.co/guide/en/cloud-on-k8s/1.7/k8s_supported_versions.html)).
I have confirmed this point, `8.x` versions for the Elastic stack are supported for `2.2+` ECK versions.
This leaves us with the 2 options I mentioned in the previous comment, let me know if you can use one of those.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 24, 2023, 9:20am UTC](https://discuss.elastic.co/t/eck-apm-server-failed-to-to-index-event-security-exception-action-indices-admin-auto-create/343741/8 "2023-10-24T09:20:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.