Hello. I'm attempting to add a 4th node to my cluster.
Nodes:
[elastic-node-01] [elastic-79] [elastic-180] Roles: master, data
[kibana-primary] Roles: ingest, ml, remote_cluster_client, transform
I'm trying to install and add Fleet to the [kibana-primary] node using the Kibana web interface, however when I go to install the elastic-agent on the node, it errors out and does not install the elastic-agent. The only output from the install process is:
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
{"log.level":"info","@timestamp":"2022-08-11T16:49:41.589Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":403},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-11T16:49:45.270Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Waiting on fleet-server input to be added to policy: fleet-server-policy","ecs.version":"1.6.0"}
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.3/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.3/fleet-troubleshooting.html
Following the troubleshooting page I tried installing the elastic-agent alone ./elastic-agent install -f, so that logging could occur, and then ran elastic-agent enroll -f -v --fleet-server-es=https://[IP ADDRESS OF elastic-node-01]:9200 --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjAyMzY0MDE3MjQ6MlZncHVUWnRSWFN6TUU3bE0wWVFLUQ --fleet-server-policy=fleet-server-policy --fleet-server-es-ca-trusted-fingerprint=07c89e04b0272a81382a73f0e5ab651e731bd1fad27349cc55948830ec2c0479
Logs:
Aug 11 17:04:10 kibana-primary elastic-agent[23454]: {"log.level":"info","@timestamp":"2022-08-11T17:04:10.308Z","log.origin":{"file.name":"operation/operator.go","file.line":222},"message":"waiting for installer of pipeline 'default' to finish","ecs.version":"1.6.0"}
Aug 11 17:04:10 kibana-primary elastic-agent[23454]: {"log.level":"info","@timestamp":"2022-08-11T17:04:10.309Z","log.origin":{"file.name":"process/app.go","file.line":182},"message":"Signaling application to stop because of shutdown: metricbeat--8.3.3--36643631373035623733363936343635","ecs.version":"1.6.0"}
Aug 11 17:04:10 kibana-primary elastic-agent[23454]: {"log.level":"info","@timestamp":"2022-08-11T17:04:10.310Z","log.origin":{"file.name":"process/app.go","file.line":182},"message":"Signaling application to stop because of shutdown: metricbeat--8.3.3","ecs.version":"1.6.0"}
Aug 11 17:04:10 kibana-primary elastic-agent[23454]: {"log.level":"info","@timestamp":"2022-08-11T17:04:10.310Z","log.origin":{"file.name":"process/app.go","file.line":182},"message":"Signaling application to stop because of shutdown: filebeat--8.3.3--36643631373035623733363936343635","ecs.version":"1.6.0"}
Aug 11 17:04:10 kibana-primary elastic-agent[23454]: {"log.level":"error","@timestamp":"2022-08-11T17:04:10.813Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"filebeat_monitoring stderr: "panic: close of closed channel
goroutine 104 [running]:
"","agent.console.name":"filebeat_monitoring","agent.console.type":"stderr","ecs.version":"1.6.0"}
Aug 11 17:04:10 kibana-primary elastic-agent[23454]: {"log.level":"error","@timestamp":"2022-08-11T17:04:10.813Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"filebeat_monitoring stderr: "github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Stop(0xc0007834a0)
/go/src/github.com/elastic/beats/filebeat/beater/filebeat.go:428 +0x3f
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch.func5()
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:461 +0x55
sync.(*Once).doSlow(0x0?, 0x0?)
/usr/local/go/src/sync/once.go:68 +0xc2
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:59
github.com/elastic/elastic-agent-libs/service.HandleSignals.func1()
/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.2.9/service/service.go:60 +0x19c
created by github.com/elastic/elastic-agent-libs/service.HandleSignals
/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.2.9/service/service.go:49 +0x18f
"","agent.console.name":"filebeat_monitoring","agent.console.type":"stderr","ecs.version":"1.6.0"}
Aug 11 17:04:10 kibana-primary elastic-agent[23454]: {"log.level":"error","@timestamp":"2022-08-11T17:04:10.833Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"metricbeat stderr: "panic: close of closed channel
goroutine 106 [running]:
github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Stop(0xc000424180?)
/go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:276 +0x1c
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch.func5()
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:461 +0x55
sync.(*Once).doSlow(0x0?, 0x0?)
/usr/local/go/src/sync/once.go:68 +0xc2
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:59
github.com/elastic/elastic-agent-libs/service.HandleSignals.func1()
/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.2.9/service/service.go:60 +0x19c
created by github.com/elastic/elastic-agent-libs/service.HandleSignals
/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.2.9/service/service.go:49 +0x18f
"","agent.console.name":"metricbeat","agent.console.type":"stderr","ecs.version":"1.6.0"}
Aug 11 17:04:10 kibana-primary elastic-agent[23454]: {"log.level":"error","@timestamp":"2022-08-11T17:04:10.833Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"metricbeat_monitoring stderr: "panic: close of closed channel
goroutine 2091 [running]:
github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Stop(0xc000244a80?)
/go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:276 +0x1c
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch.func5()
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:461 +0x55
sync.(*Once).doSlow(0x0?, 0x0?)
/usr/local/go/src/sync/once.go:68 +0xc2
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:59
github.com/elastic/elastic-agent-libs/service.HandleSignals.func1()
/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.2.9/service/service.go:60 +0x19c
created by github.com/elastic/elastic-agent-libs/service.HandleSignals
/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.2.9/service/service.go:49 +0x18f
"","agent.console.name":"metricbeat_monitoring","agent.console.type":"stderr","ecs.version":"1.6.0"}
Aug 11 17:04:10 kibana-primary elastic-agent[23454]: {"log.level":"info","@timestamp":"2022-08-11T17:04:10.842Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-08-11T17:04:10Z - message: Application: metricbeat--8.3.3--36643631373035623733363936343635[81dd967a-0dea-4eab-ab3f-f0e7066994af]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
Aug 11 17:04:11 kibana-primary elastic-agent[23454]: {"log.level":"info","@timestamp":"2022-08-11T17:04:11.822Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-08-11T17:04:11Z - message: Application: filebeat--8.3.3--36643631373035623733363936343635[81dd967a-0dea-4eab-ab3f-f0e7066994af]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
Aug 11 17:04:11 kibana-primary elastic-agent[23454]: {"log.level":"info","@timestamp":"2022-08-11T17:04:11.843Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-08-11T17:04:11Z - message: Application: metricbeat--8.3.3[81dd967a-0dea-4eab-ab3f-f0e7066994af]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
At this point I'm not sure where else to check for errors. The cluster itself reports that it's healthy. And there don't appear to be security/certificate errors during the enroll.
Any advice on how to proceed would be appreciated.