Hello,
I'm configuring APM behind reverse proxy(nginx), Successfully created fleet server and enrolled elastic agent. But the APM server is not working.
Docker version: Docker version 20.10.17, build 100c701
Elastic agent image: docker.elastic.co/beats/elastic-agent:8.2.0
{"log.level":"info","@timestamp":"2022-07-26T04:05:27.191Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:27Z - message: Application: apm-server--8.2.0[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to CONFIG: Updating configuration - type: 'STATE' - sub_type: 'CONFIG'","ecs.version":"1.6.0"}
Elastic agent docker container log:
{"log.level":"warn","@timestamp":"2022-07-26T04:04:21.690Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":105},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:04:22.375Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Starting enrollment to URL: http://fleet-server:8220/","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-07-26T04:04:22.477Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":472},"message":"Remote server is not ready to accept connections, will retry in a moment.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:22.478Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":479},"message":"Retrying enrollment to URL: http://fleet-server:8220/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.236Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":259},"message":"Elastic Agent might not be running; unable to trigger restart","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.342Z","log.origin":{"file.name":"cmd/run.go","file.line":153},"message":"APM instrumentation disabled","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.343Z","log.origin":{"file.name":"application/application.go","file.line":78},"message":"Detecting execution mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.344Z","log.origin":{"file.name":"application/application.go","file.line":102},"message":"Agent is managed by Fleet","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.344Z","log.origin":{"file.name":"capabilities/capabilities.go","file.line":59},"message":"capabilities file not found in /usr/share/elastic-agent/state/capabilities.yml","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-07-26T04:05:23.344Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":105},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.847Z","log.logger":"composable.providers.docker","log.origin":{"file.name":"docker/docker.go","file.line":43},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.848Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":63},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.848Z","log.origin":{"file.name":"application/managed_mode.go","file.line":299},"message":"Agent is starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:23.848Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":65},"message":"Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/elastic-agent.sock)","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-07-26T04:05:25.143Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":105},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:25.165Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is UXcPYvJ6","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:25.166Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 4 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:25.545Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for metricbeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:25.676Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:25Z - message: Application: metricbeat--8.2.0[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:25.877Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for apm-server.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:26.088Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:26Z - message: Application: apm-server--8.2.0[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:26.396Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for filebeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:26.722Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:26Z - message: Application: filebeat--8.2.0[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:27.044Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for filebeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:27.191Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:27Z - message: Application: apm-server--8.2.0[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to CONFIG: Updating configuration - type: 'STATE' - sub_type: 'CONFIG'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:27.265Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:27Z - message: Application: filebeat--8.2.0--36643631373035623733363936343635[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:27.652Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for metricbeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:27.755Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:27Z - message: Application: metricbeat--8.2.0--36643631373035623733363936343635[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:27.761Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:29.890Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:29Z - message: Application: metricbeat--8.2.0[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:30.960Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:30Z - message: Application: filebeat--8.2.0[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:31.469Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:31Z - message: Application: filebeat--8.2.0--36643631373035623733363936343635[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-26T04:05:31.957Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-07-26T04:05:31Z - message: Application: metricbeat--8.2.0--36643631373035623733363936343635[b7c2f69b-e7e8-48e0-906e-f72d19fe86d9]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
Inside the container:
root@ce7b94b0b022:/usr/share/elastic-agent# elastic-agent status
Status: HEALTHY
Message: (no message)
Applications:
* metricbeat_monitoring (HEALTHY)
Running
* metricbeat (HEALTHY)
Running
* apm-server (CONFIGURING)
Updating configuration
* filebeat (HEALTHY)
Running
* filebeat_monitoring (HEALTHY)
Running
Would appreciate any help