Dockerize Elastic Agent APM server: Stuck on CONFIG: Updating configuration

Elastic Observability APM
, ,
1

Hello,

I'm configuring APM behind reverse proxy(nginx), Successfully created fleet server and enrolled elastic agent. But the APM server is not working.

Docker version: Docker version 20.10.17, build 100c701
Elastic agent image: docker.elastic.co/beats/elastic-agent:8.2.0

{"log.level":"info","@timestamp":"2022-07-26T04:05:27.191Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:27Z - message: Application: apm-server--8.2.0[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to CONFIG: Updating configuration - type: 'STATE' - sub_type: 'CONFIG'","ecs.version":"1.6.0"}

Elastic agent docker container log:

{"log.level":"warn","@timestamp":"2022-07-26T04:04:21.690Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":105},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:04:22.375Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Starting enrollment to URL: http://fleet-server:8220/","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-07-26T04:04:22.477Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":472},"message":"Remote server is not ready to accept connections, will retry in a moment.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:22.478Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":479},"message":"Retrying enrollment to URL: http://fleet-server:8220/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.236Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":259},"message":"Elastic Agent might not be running; unable to trigger restart","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.342Z","log.origin":{"file.name":"cmd/run.go","file.line":153},"message":"APM instrumentation disabled","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.343Z","log.origin":{"file.name":"application/application.go","file.line":78},"message":"Detecting execution mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.344Z","log.origin":{"file.name":"application/application.go","file.line":102},"message":"Agent is managed by Fleet","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.344Z","log.origin":{"file.name":"capabilities/capabilities.go","file.line":59},"message":"capabilities file not found in /usr/share/elastic-agent/state/capabilities.yml","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-07-26T04:05:23.344Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":105},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.847Z","log.logger":"composable.providers.docker","log.origin":{"file.name":"docker/docker.go","file.line":43},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.848Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":63},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.848Z","log.origin":{"file.name":"application/managed_mode.go","file.line":299},"message":"Agent is starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.848Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":65},"message":"Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/elastic-agent.sock)","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-07-26T04:05:25.143Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":105},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:25.165Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is UXcPYvJ6","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:25.166Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 4 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:25.545Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for metricbeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:25.676Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:25Z - message: Application: metricbeat--8.2.0[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:25.877Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for apm-server.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:26.088Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:26Z - message: Application: apm-server--8.2.0[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:26.396Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for filebeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:26.722Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:26Z - message: Application: filebeat--8.2.0[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:27.044Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for filebeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:27.191Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:27Z - message: Application: apm-server--8.2.0[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to CONFIG: Updating configuration - type: 'STATE' - sub_type: 'CONFIG'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:27.265Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:27Z - message: Application: filebeat--8.2.0--36643631373035623733363936343635[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:27.652Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for metricbeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:27.755Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:27Z - message: Application: metricbeat--8.2.0--36643631373035623733363936343635[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:27.761Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:29.890Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:29Z - message: Application: metricbeat--8.2.0[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:30.960Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:30Z - message: Application: filebeat--8.2.0[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:31.469Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:31Z - message: Application: filebeat--8.2.0--36643631373035623733363936343635[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:31.957Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:31Z - message: Application: metricbeat--8.2.0--36643631373035623733363936343635[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}

Inside the container:

root@ce7b94b0b022:/usr/share/elastic-agent# elastic-agent status
Status: HEALTHY
Message: (no message)
Applications:
  * metricbeat_monitoring  (HEALTHY)
                           Running
  * metricbeat             (HEALTHY)
                           Running
  * apm-server             (CONFIGURING)
                           Updating configuration
  * filebeat               (HEALTHY)
                           Running
  * filebeat_monitoring    (HEALTHY)
                           Running

Would appreciate any help

2

Hello @Orville_Lim ,

Can you share the full apm-server logs to help in debugging? The default location would be /usr/share/elastic-agent/state/data/logs/default/apm-server-*.ndjson inside the agent container.

1 Like
3

Hi @lahsivjar ,

Thanks for replying it's solved, I configured wrong host in APM integration config in Kibana.

4

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.