# APM Server 9.1.4 Container Always Fails with 401 Despite Valid Credentials
**URL:** https://discuss.elastic.co/t/apm-server-9-1-4-container-always-fails-with-401-despite-valid-credentials/382139
**Category:** APM
**Tags:** docker, server
**Created:** [September 22, 2025, 11:14pm UTC](https://discuss.elastic.co/t/apm-server-9-1-4-container-always-fails-with-401-despite-valid-credentials/382139 "2025-09-22T23:14:10Z")
**Posts on this page:** 1
**Page:** 1
### Author:  [@Andrey\_Smirnov](https://discuss.elastic.co/u/Andrey_Smirnov)
#### Post date: [September 22, 2025, 11:14pm UTC](https://discuss.elastic.co/t/apm-server-9-1-4-container-always-fails-with-401-despite-valid-credentials/382139/1 "2025-09-22T23:14:10Z")
I’m running Elasticsearch, Kibana, and APM Server 9.1.4 in Docker using Docker Compose. All containers are on the same custom bridge network. Elasticsearch shows `green` cluster health and the `apm_system` user is enabled with the correct password.
Example checks from the host and Elasticsearch container confirm the credentials work:
```auto
curl -u apm_system: http://localhost:9200/_cluster/health?pretty
# Returns cluster health GREEN
```
However, the APM Server container logs repeatedly show:
```auto
precondition failed: error querying cluster_uuid: status_code=401
refresh cache error: context deadline exceeded
```
- The container is minimal — it doesn’t have `curl`, `ping`, or `sh`, so I cannot test connectivity from inside.
- I verified environment variables are set correctly in the container:
```auto
APM_ELASTIC_USERNAME=apm_system
APM_ELASTIC_PASSWORD=
```
- Kibana is not involved in this setup (no Fleet).
- Elasticsearch is reachable from host and other containers.
It seems like the APM Server container cannot authenticate with Elasticsearch, even though credentials are correct. Could this be a networking or minimal container issue? Any suggestions for debugging or a reliable way to confirm connectivity from inside the APM container?
## **Additional Context and Findings**
**Updated Configuration Attempts:**
- Created a custom apm\_writer user with apm\_writer\_role that has full permissions for APM indices (apm-\*, traces-\*, metrics-apm-\*, logs-apm-\*) including create\_index, write, manage privileges
- Updated APM Server environment variables to use the new user: APM\_ELASTIC\_USERNAME=apm\_writer and APM\_ELASTIC\_PASSWORD=apm\_writer\_password123!
- Added cluster-level permissions (monitor, cluster:admin/xpack/monitoring/bulk) to the custom role
- Verified the custom user can successfully create indices: curl -u apm\_writer:apm\_writer\_password123! -X PUT "localhost:9200/apm-test-index" returns {"acknowledged":true}
**Docker Compose Configuration:**
apm-server:
environment:
```
- output.elasticsearch.hosts=\["http://elasticsearch:9200"\]
- output.elasticsearch.username=${APM_ELASTIC_USERNAME}
- output.elasticsearch.password=${APM_ELASTIC_PASSWORD}
- apm-server.host=0.0.0.0:8200
- apm-server.data_streams.wait_for_integration=false
```
**Current Behavior:**
- APM Server accepts requests from .NET applications (returns 202 status codes)
- No APM data indices are being created in Elasticsearch
- APM Server remains in "blocking ingestion until all preconditions are satisfied" state
- The 401 errors persist even with a user that has full cluster and index permissions
**Key Questions:**
1. In Elasticsearch 9.1.4, is the apm\_system user still the recommended approach, or should we use a custom user with full permissions?
2. Are there specific cluster-level permissions required for APM Server beyond monitor and cluster:admin/xpack/monitoring/bulk?
3. Could this be related to the APM Server's internal authentication mechanism or a known issue with the 9.1.4 version?