# Agent configuration infrastructure is not ready

**URL:** <https://discuss.elastic.co/t/agent-configuration-infrastructure-is-not-ready/348364>\
**Category:** APM\
**Tags:** docker, nodejs, server\
**Created:** [November 30, 2023, 7:40pm UTC](https://discuss.elastic.co/t/agent-configuration-infrastructure-is-not-ready/348364 "2023-11-30T19:40:01Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![joe\_recra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_recra/32/128281_2.png) [@joe\_recra](https://discuss.elastic.co/u/joe_recra)\
**Post date:** [November 30, 2023, 7:40pm UTC](https://discuss.elastic.co/t/agent-configuration-infrastructure-is-not-ready/348364/1 "2023-11-30T19:40:01Z")

</div>

**Kibana version** :  
8.11.1  
**Elasticsearch version** :  
8.11.1  
**APM Server version** :  
8.11.1  
**APM Agent language and version** :  
Nodejs v17.9.1 && elastic-apm-node 4.2.0  
**Browser version** :

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**:  
Docker-compose  
**Fresh install or upgraded from other version?**

**Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:

I configured ES, KIBANA and APM-SERVER with docker-compose. When Im trying to run app it throws: "APM Server transport error (503): Unexpected APM Server response when polling config\nAgent configuration infrastructure is not ready. Please retry later."

**Steps to reproduce** :

1. Nodejs code:

```auto
var apm = require("elastic-apm-node").start({
  serviceName: "my-service-name",
  secretToken: "",
  serverUrl: "http://127.0.0.1:8200",
  environment: "my-environment",
});

```

1. node index.js

**Errors in browser console (if relevant)**:

**Provide logs and/or server output (if relevant)**:

Nodejs Express app logs:

`{"log.level":"info","@timestamp":"2023-11-30T19:20:54.100Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","agentVersion":"4.2.0","env":{"pid":31404,"proctitle":"node","os":"linux 4.15.0-213-generic","arch":"x64","host":"joerecra-HP-ProBook-450-G5","timezone":"UTC-0500","runtime":"Node.js v14.15.1"},"config":{"environment":{"source":"start","value":"my-environment"},"logLevel":{"source":"default","value":"info","commonName":"log_level"},"serverUrl":{"source":"start","value":"http://127.0.0.1:8200/","commonName":"server_url"},"secretToken":{"source":"start","value":"[REDACTED]","commonName":"secret_token"},"serviceName":{"source":"start","value":"my-service-name","commonName":"service_name"},"serviceVersion":{"source":"default","value":"1.0.0","commonName":"service_version"}},"activationMethod":"require","message":"Elastic APM Node.js Agent v4.2.0"} {"log.level":"error","@timestamp":"2023-11-30T19:20:54.220Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","message":"APM Server transport error (503): Unexpected APM Server response when polling config\nAgent configuration infrastructure is not ready. Please retry later."}`

`curl -v -X POST http://localhost:8200/`

```auto
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8200 (#0)
> POST / HTTP/1.1
> Host: localhost:8200
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< X-Content-Type-Options: nosniff
< Date: Thu, 30 Nov 2023 19:28:34 GMT
< Content-Length: 151
< 
{
  "build_date": "2023-11-07T18:53:29Z",
  "build_sha": "f0dee4830530d39e136f3e88ee45ab20cfacc241",
  "publish_ready": false,
  "version": "8.11.1"
}

```

```auto
{"log.level":"info","@timestamp":"2023-11-30T19:17:20.728Z","log.origin":{"file.name":"beatcmd/beat.go","file.line":139},"message":"Home path: [/usr/share/apm-server] Config path: [/usr/share/apm-server] Data path: [/usr/share/apm-server/data] Logs path: [/usr/share/apm-server/logs]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-30T19:17:20.728Z","log.origin":{"file.name":"beatcmd/beat.go","file.line":146},"message":"Beat ID: fcaa31e3-06d9-4b52-bb83-8961b6b86b7d","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-30T19:17:20.728Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":122},"message":"elasticsearch url: https://elasticsearch:9200","service.name":"apm-server","ecs.version":"1.6.0"}

```

Some apm-server logs errors:

```auto
{"log.level":"info","@timestamp":"2023-11-30T19:26:34.143Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/4.2.0 (my-service-name 1.0.0)","source.address":"192.168.224.1","http.request.id":"bcd0cca7-4f70-459b-b314-66cb4f6769b9","event.duration":10001863299,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-30T19:26:36.875Z","log.logger":"agentcfg","log.origin":{"file.name":"agentcfg/elasticsearch.go","file.line":150},"message":"refresh cache error: context deadline exceeded","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-11-30T19:26:44.518Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 30, 2023, 7:58pm UTC](https://discuss.elastic.co/t/agent-configuration-infrastructure-is-not-ready/348364/2 "2023-11-30T19:58:36Z")

</div>

> [@joe\_recra](#):
>
> I configured ES, KIBANA and APM-SERVER with docker-compose. When Im trying to run app it throws: "APM Server transport error (503): Unexpected APM Server response when polling config\nAgent configuration infrastructure is not ready. Please retry later."

> [@joe\_recra](#):
>
> ```auto
> {"log.level":"error","@timestamp":"2023-11-30T19:26:36.875Z","log.logger":"agentcfg","log.origin":{"file.name":"agentcfg/elasticsearch.go","file.line":150},"message":"refresh cache error: context deadline exceeded","service.name":"apm-server","ecs.version":"1.6.0"}
> {"log.level":"error","@timestamp":"2023-11-30T19:26:44.518Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
> 
> ```

It probably means your APM server can not connect to elasticsearch... probably because your elasticsearch has a self-signed cert and thus the APM server fails to connect.

See [SSL/TLS output settings | APM User Guide [8.11] | Elastic](https://www.elastic.co/guide/en/apm/guide/current/configuration-ssl.html)

You can share your compose

---

<div class="post-metadata">

**Author:** ![joe\_recra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_recra/32/128281_2.png) [@joe\_recra](https://discuss.elastic.co/u/joe_recra)\
**Post date:** [November 30, 2023, 8:17pm UTC](https://discuss.elastic.co/t/agent-configuration-infrastructure-is-not-ready/348364/3 "2023-11-30T20:17:34Z")

</div>

thank you for your quick response, my docker-compose file

```auto
version: "3"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.1
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
    ports:
      - "9200:9200"
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: 2056M

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.1
    container_name: kibana
    ports:
      - "5601:5601"
    environment:
      ELASTICSEARCH_URL: http://elasticsearch:9200
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    depends_on:
      - elasticsearch

  apm:
    image: docker.elastic.co/apm/apm-server:8.11.1
    container_name: apm-server
    ports:
      - "8200:8200"
    volumes:
      - ./apm-server.yml:/usr/share/apm-server/apm-server.yml
      - ./var/log/apm-server:/var/log/apm-server

    depends_on:
      - elasticsearch

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 30, 2023, 8:21pm UTC](https://discuss.elastic.co/t/agent-configuration-infrastructure-is-not-ready/348364/4 "2023-11-30T20:21:48Z")

</div>

Kinda surprised any of that is working.....

Can you log into Elastic Via Kibana?

I do not think this is correct elasticsearch unless you did something should be running with security... did you manually change setting in the elasticsearch.yml  
Hmm not sure what you really did?

`ELASTICSEARCH_URL: http://elasticsearch:9200`

Can you curl elasticsearch? from either your host or from the Kibana or APM container?

I guess in short elastic 8.11.1 configures security by default ... or tries to and I do not see any of that in your compose...

---

<div class="post-metadata">

**Author:** ![joe\_recra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_recra/32/128281_2.png) [@joe\_recra](https://discuss.elastic.co/u/joe_recra)\
**Post date:** [November 30, 2023, 10:43pm UTC](https://discuss.elastic.co/t/agent-configuration-infrastructure-is-not-ready/348364/5 "2023-11-30T22:43:03Z")

</div>

when I execute:  
`curl https://localhost:9200/`  
response:  
`curl: (60) SSL certificate problem: self signed certificate in certificate chain`

but when I run:  
`curl -k https://localhost:9200/`  
response:  
`{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}`

I think when I entered to Kibana, and it asked for me the enroll secret and login, it installed some ssl, because my kibana.yml config changed and initial config were commented by system:

Kibana.yml

```auto
### >>>>>>> BACKUP START: Kibana interactive setup (2023-11-30T19:09:38.898Z)

#Kibana server host

#server.host: 0.0.0.0

#Elasticsearch host

#elasticsearch.hosts: ["http://localhost:9200"]

#xpack.security.enabled: true

#xpack.fleet.enabled: true

#xpack.fleet.agents.enabled: true

#xpack.encryptedSavedObjects.encryptionKey: "43073011488510979632349040114752"

### >>>>>>> BACKUP END: Kibana interactive setup (2023-11-30T19:09:38.898Z)

# This section was automatically generated during setup.

server.host: 0.0.0.0

elasticsearch.hosts: ['https://192.168.224.2:9200']

xpack.security.enabled: true

xpack.fleet.enabled: true

xpack.fleet.agents.enabled: true

xpack.encryptedSavedObjects.encryptionKey: '43073011488510979632349040114752'

elasticsearch.serviceAccountToken: AAEAAWVsYXN0aWMva2liYW5hL2Vucm9sbC1wcm9jZXNzLXRva2VuLTE3MDEzNzEzNzc3NTA6NVJuUmFsWTJUZXlraG01MkNiU01sUQ

elasticsearch.ssl.certificateAuthorities: [/usr/share/kibana/data/ca_1701371378896.crt]

xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://192.168.224.2:9200'], ca_trusted_fingerprint: 27e5d9a0a78c224f2e45bf94dc6e6510180f1e78eb1654f2432c16f082dffed4}]

```

this is my apm-server config:

```auto
apm-server:
  host: "0.0.0.0:8200"
output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  protocol: "https"
  username: "elastic"
  password: "ZGhS8vFS19ehbZuTRQ60"
# Logging settings
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/apm-server
  name: apm-server
  keepfiles: 7
  permissions: 0644

```

Then, What do I need to do?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 30, 2023, 11:13pm UTC](https://discuss.elastic.co/t/agent-configuration-infrastructure-is-not-ready/348364/6 "2023-11-30T23:13:42Z")

</div>

OK so you did do some configuration ... good...

> [@joe\_recra](#):
>
> curl -k [https://localhost:9200/](https://localhost:9200/)

run

`curl -k -v -u elastic https://localhost:9200/`

and enter the elastic user password

If that works, you have 2 options to get the APM server working that I referenced above.

```auto
output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  protocol: "https"
  username: "elastic"
  password: "ZGhS8vFS19ehbZuTRQ60"
  ssl.verification_mode: "none"

```

> ###### `verification_mode`
> 
> Controls the verification of server certificates. Valid values are:  
> ...  
> `none`
> 
> Performs _no verification_ of the server’s certificate. This mode disables many of the security benefits of SSL/TLS and should only be used after cautious consideration. It is primarily intended as a temporary diagnostic mechanism when attempting to resolve TLS errors; its use in production environments is strongly discouraged.

Or you can get the CA from the elasticsearch or kibana containers copy it in the APM container and reference... see [here](https://www.elastic.co/guide/en/apm/guide/current/configuration-ssl.html#client-certificate-authorities)

the verification mode none is probably fine if this is dev / test

---

<div class="post-metadata">

**Author:** ![joe\_recra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_recra/32/128281_2.png) [@joe\_recra](https://discuss.elastic.co/u/joe_recra)\
**Post date:** [December 1, 2023, 6:13pm UTC](https://discuss.elastic.co/t/agent-configuration-infrastructure-is-not-ready/348364/7 "2023-12-01T18:13:03Z")

</div>

Thank you so much, it worked. Then, I'll configure elastic with cert in docker-compose.

off topic: Is there any way to log from aws lambda to elasticsearch without passing though Cloudwatch, I mean without using functionBeats or SAR Forwader ?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 1, 2023, 7:05pm UTC](https://discuss.elastic.co/t/agent-configuration-infrastructure-is-not-ready/348364/8 "2023-12-01T19:05:50Z")

</div>

> [@joe\_recra](#):
>
> off topic: Is there any way to log from aws lambda to elasticsearch without passing though Cloudwatch, I mean without using functionBeats or SAR Forwader ?

Open a new topic.. through kinesis... there is only so many options 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 29, 2023, 7:06pm UTC](https://discuss.elastic.co/t/agent-configuration-infrastructure-is-not-ready/348364/9 "2023-12-29T19:06:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.