Need help on fleet
I have recently installed a cluster (8.4.0 docker) and facing issues with fleet/APM integration.
Fleet container is running in the same elk server and getting registered with healthy status. However, unable to view APM data/fleet ->data streams/fleet ->agent-> logs. I believe fleet is unable to push the data to ES.
I tried to troubleshoot but unable to find much info.
Docker-compose file:
version: '3.5'
services:
fleet:
image: "docker.elastic.co/beats/elastic-agent-complete:8.4.0"
container_name: "elk-fleet"
hostname: "elk-fleet"
user: root
environment:
- FLEET_SERVER_ENABLE=true
- FLEET_SERVER_ELASTICSEARCH_HOST=https://elkserverfqdn:9200
- FLEET_SERVER_POLICY_ID=fleet-server-policy(created with wizad)
- FLEET_SERVER_SERVICE_TOKEN=zzzzzzzzzzzzzzzzzzzzzzzzzz
- FLEET_URL=https://elkserverfqdn:8220
- FLEET-SERVER-CERT-CA= cacert.pem
- FLEET_SERVER_CERT= cert.pem
- FLEET_SERVER_CERT_KEY= cert.pem
- FLEET_SERVER_ELASTICSEARCH_CA_TRUSTED_FINGERPRINT=base64ofcasha256fingerprint
- FLEET_SERVER_ELASTICSEARCH_CA= cacert.pem
- FLEET_CA=cacert.pem
ports:
- 8200:8200
- 8220:8220
(tried insecure options too - FLEET_SERVER_ELASTICSEARCH_INSECURE=true and FLEET_SERVER_INSECURE_HTTP=true)
There is only one policy and with fleet server, system & Elastic APM integrations with collection of logs, metrics enabled. However, logs are not observed. Also changed logging level to “debug”.
Fleet Settings:
Fleet server hosts:
https://elkserverfqdn:8220 (fleet is on elk server)
outputs
https://elkserverfqdn:9200
ca fingerprint: base64of_ca_sha256fingerprint
Fleet config:
Host: 0.0.0.0 port 8200
APM:
Host: 0.0.0.0:8200
url: http://elkserverfqdn:8200
RUM : enabled
Agent auth Anonymous enabled; allowed agents “*” (added)
Errors found from fleet container
{"log":"{"log.level":"error","@timestamp":"2022-10-12T18:22:01.487Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"fleet-server stderr: \"{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2022-10-12T18:22:01Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2022-10-12T18:22:01Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n\"","agent.console.name":"fleet-server","agent.console.type":"stderr","ecs.version":"1.6.0"}\n","stream":"stderr","time":"2022-10-12T18:22:01.487682391Z"}
{"log":"{"log.level":"error","@timestamp":"2022-10-12T18:22:05.889Z","log.origin":{"file.name":"process/app.go","file.line":290},"message":"failed to stop fleet-server: os: process already finished","ecs.version":"1.6.0"}\n","stream":"stderr","time":"2022-10-12T18:22:05.889309437Z"}
{"log":"{"log.level":"error","@timestamp":"2022-10-12T18:22:08.044Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"fleet-server stderr: \"{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2022-10-12T18:22:08Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2022-10-12T18:22:08Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n\"","agent.console.name":"fleet-server","agent.console.type":"stderr","ecs.version":"1.6.0"}\n","stream":"stderr","time":"2022-10-12T18:22:08.045445124Z"}
{"log":"{"log.level":"error","@timestamp":"2022-10-12T18:22:14.556Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"filebeat_monitoring stderr: \"panic: close of closed channel\\n\\ngoroutine 159 [running]:\\ngithub.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Stop(0xc00072f620\"","agent.console.name":"filebeat_monitoring","agent.console.type":"stderr","ecs.version":"1.6.0"}\n","stream":"stderr","time":"2022-10-12T18:22:14.55663006Z"}
{"log":"{"log.level":"error","@timestamp":"2022-10-12T18:22:14.556Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"filebeat_monitoring stderr: \")\\n\\t/go/src/github.com/elastic/beats/filebeat/beater/filebeat.go:428 +0x46\\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch.func5()\\n\\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:461 +0x68\\nsync.(*Once).doSlow(0xc000647980, 0xc00095b340)\\n\\t/usr/local/go/src/sync/once.go:68 +0x178\\nsync.(*Once).Do(0xc000647980, 0xc00095b340)\\n\\t/usr/local/go/src/sync/once.go:59 +0x45\\ngithub.com/elastic/elastic-agent-libs/service.HandleSignals.func1()\\n\\t/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.2.9/service/service.go:60 +0x20c\\ncreated by github.com/elastic/elastic-agent-libs/service.HandleSignals\\n\\t/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.2.9/service/service.go:49 +0x268\\n\"","agent.console.name":"filebeat_monitoring","agent.console.type":"stderr","ecs.version":"1.6.0"}\n","stream":"stderr","time":"2022-10-12T18:22:14.556670107Z"}
{"log":"{"log.level":"error","@timestamp":"2022-10-12T18:22:15.504Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"metricbeat_monitoring stderr: \"panic: close of closed channel\\n\\ngoroutine 121 [running]:\\ngithub.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Stop(0xc0009fa1c0)\\n\\t/go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:276 +0x28\\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch.func5()\\n\\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:461 +0x68\\nsync.(*Once).doSlow(0xc0005e5110, 0xc0013de680)\\n\\t/usr/local/go/src/sync/once.go:68 +0x178\\nsync.(*Once).Do(0xc0005e5110, 0xc0013de680)\\n\\t/usr/local/go/src/sync/once.go:59 +0x45\\ngithub.com/elastic/elastic-agent-libs/service.HandleSignals.func1()\\n\\t/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.2.9/service/service.go:60 +0x20c\\ncreated by github.com/elastic/elastic-agent-libs/service.HandleSignals\\n\\t/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.2.9/service/service.go:49 +0x268\\n\"","agent.console.name":"metricbeat_monitoring","agent.console.type":"stderr","ecs.version":"1.6.0"}\n","stream":"stderr","time":"2022-10-12T18:22:15.504644325Z"}
Am I missing something or configuring incorrectly? pls suggest.