Trouble setting up Elastic Agent Fleet server on self-managed Elasticsearch setup

Elastic Stack Beats
,
1

hi all! I'm having a hard time setting up the elastic agent fleet server. The environment: single node Elasticsearch/wazuh installation with filebeat/metricbeat/heartbeat installed on the node host. I used to copy & paste code from the walk through on the Fleet page on my node host & this is the terminal output:

root@monitor3:/etc/elasticsearch/elastic-agent/elastic-agent-7.14.2-linux-x86_64# ./elastic-agent install --fleet-server-es=http://localhost:9200 --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2Mzg0NjQ4MzczMDQ6WXVQU3FyX0tTX09uSjdoa29WUU5nQQ --fleet-server-policy=7b1bb540-521f-11ec-a758-410e5d263baa
Elastic Agent is installed but currently broken: service is not installed
Continuing will re-install Elastic Agent over the current installation at /opt/Elastic/Agent. Do you want to continue? [Y/n]:y
2021-12-02T12:47:53.865-0500    INFO    cmd/enroll_cmd.go:336   Generating self-signed certificate for Fleet Server
2021-12-02T12:47:55.733-0500    INFO    cmd/enroll_cmd.go:650   Waiting for Elastic Agent to start Fleet Server
2021-12-02T12:47:57.736-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Starting
2021-12-02T12:47:58.738-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Error - EOF
2021-12-02T12:48:04.745-0500    INFO    cmd/enroll_cmd.go:688   Fleet Server - Error - EOF
2021-12-02T12:48:08.750-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Restarting
2021-12-02T12:48:09.752-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Error - EOF
2021-12-02T12:48:15.761-0500    INFO    cmd/enroll_cmd.go:688   Fleet Server - Error - EOF
2021-12-02T12:48:19.768-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Restarting
2021-12-02T12:48:20.769-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Error - EOF
2021-12-02T12:48:26.778-0500    INFO    cmd/enroll_cmd.go:688   Fleet Server - Error - EOF
2021-12-02T12:48:30.784-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Restarting
2021-12-02T12:48:31.786-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Error - EOF
2021-12-02T12:48:37.795-0500    INFO    cmd/enroll_cmd.go:688   Fleet Server - Error - EOF
2021-12-02T12:48:41.800-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Restarting
2021-12-02T12:48:42.802-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Error - EOF
2021-12-02T12:48:48.811-0500    INFO    cmd/enroll_cmd.go:688   Fleet Server - Error - EOF
2021-12-02T12:48:52.828-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Restarting
2021-12-02T12:48:53.829-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Error - EOF
2021-12-02T12:48:59.837-0500    INFO    cmd/enroll_cmd.go:688   Fleet Server - Error - EOF
2021-12-02T12:49:03.843-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Restarting
2021-12-02T12:49:04.845-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Error - EOF
2021-12-02T12:49:10.853-0500    INFO    cmd/enroll_cmd.go:688   Fleet Server - Error - EOF
2021-12-02T12:49:14.859-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Restarting
2021-12-02T12:49:15.861-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Error - EOF
2021-12-02T12:49:21.870-0500    INFO    cmd/enroll_cmd.go:688   Fleet Server - Error - EOF
2021-12-02T12:49:25.877-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Restarting
2021-12-02T12:49:26.878-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Error - EOF
2021-12-02T12:49:32.887-0500    INFO    cmd/enroll_cmd.go:688   Fleet Server - Error - EOF
2021-12-02T12:49:36.893-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Restarting
2021-12-02T12:49:37.895-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Error - EOF
2021-12-02T12:49:43.904-0500    INFO    cmd/enroll_cmd.go:688   Fleet Server - Error - EOF
2021-12-02T12:49:47.910-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Restarting
2021-12-02T12:49:48.911-0500    INFO    cmd/enroll_cmd.go:683   Fleet Server - Error - EOF
Error: fleet-server never started by elastic-agent daemon: context canceled
Error: enroll command failed with exit code: 1

Here is the syslog output:

Dec  2 12:47:52 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:52.976-0500#011INFO#011application/application.go:67#011Detecting execution mode
Dec  2 12:47:52 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:52.977-0500#011INFO#011application/application.go:76#011Agent is managed locally
Dec  2 12:47:52 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:52.977-0500#011INFO#011capabilities/capabilities.go:59#011capabilities file not found in /opt/Elastic/Agent/capabilities.yml
Dec  2 12:47:53 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:53.452-0500#011INFO#011[composable.providers.docker]#011docker/docker.go:43#011Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Dec  2 12:47:53 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:53.452-0500#011INFO#011[api]#011api/server.go:62#011Starting stats endpoint
Dec  2 12:47:53 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:53.452-0500#011INFO#011application/local_mode.go:168#011Agent is starting
Dec  2 12:47:53 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:53.452-0500#011INFO#011[api]#011api/server.go:64#011Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
Dec  2 12:47:53 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:53.453-0500#011INFO#011application/local_mode.go:178#011Agent is stopped
Dec  2 12:47:53 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:53.453-0500#011INFO#011application/periodic.go:79#011Configuration changes detected
Dec  2 12:47:53 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:53.458-0500#011INFO#011stateresolver/stateresolver.go:48#011New State ID is LGf1EY3w
Dec  2 12:47:53 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:53.458-0500#011INFO#011stateresolver/stateresolver.go:49#011Converging state requires execution of 2 step(s)
Dec  2 12:47:54 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:54.731-0500#011INFO#011operation/operator.go:192#011waiting for installer of pipeline 'default' to finish
Dec  2 12:47:55 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:55.869-0500#011ERROR#011status/reporter.go:236#011Elastic Agent status changed to: 'error'
Dec  2 12:47:55 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:55.869-0500#011ERROR#011log/reporter.go:36#0112021-12-02T12:47:55-05:00 - message: Application: metricbeat--7.14.2[2738cdde-f9b4-4f60-b1e4-729545c227a1]: State changed to FAILED: context canceled - type: 'ERROR' - sub_type: 'FAILED'
Dec  2 12:47:55 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:55.869-0500#011INFO#011process/app.go:176#011Signaling application to stop because of shutdown: metricbeat--7.14.2
Dec  2 12:47:55 monitor3 elastic-agent[3556576]: 2021-12-02T12:47:55.869-0500#011INFO#011log/reporter.go:40#0112021-12-02T12:47:55-05:00 - message: Application: metricbeat--7.14.2[2738cdde-f9b4-4f60-b1e4-729545c227a1]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'

The most confusing thing about this is that it seems related to metricbeat (see the metricbeat error in the syslog output). It throws the same error even when metricbeat is stopped.

2

Hi @iamuser I must confess I have not dig fully into this yet but looking at the fleet-server logs with the EOF error directly reminds me of a bug we had in 7.14. Any chance you could run 7.15 or even better 7.16?

3

7.14 is the highest I can go right now because we're using it with Wazuh. Do you have any troubleshooting ideas I can try out?

4

There is unfortunately not really a workaround for the EOF issue. But it is a bit surprising that you see it constantly, often it is only seen from time to time.

Maybe we should also dig into the metricbeat error. Can you check the metricbeat logs under /data/logs/...* and share it here?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.