Logs are not received on the Fleet server

b.kaliuzhnyi · January 25, 2022, 12:14pm

The situation is similar to that of the link:

https://discuss.elastic.co/t/elastic-agent-error/282706

I have installed two Elastic Agents, one of them as a Fleet server. Their status is Healthy. The fleet server policy is configured on the Fleet Server, and the Windows and Endpoint policy is configured on the other. Events don't come from both, not even metric events.

I set up encryption on the fleet server with the --insecure key, created keys and certificates with the same CA that I encrypted Elasticsearch with. I don't know, it may be related to this...

On both agents, when entering the "elastic-agent status" command, the following message is displayed:

"Error: failed to communicate with Elastic Agent daemon: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/lib/elastic-agent/data/tmp/elastic-agent-control.sock: connect: permission denied""

Logs on the sudo journalctl --unit elastic-agent > 1.txt will be attached below.

b.kaliuzhnyi · January 25, 2022, 12:17pm

-- Journal begins at Wed 2021-10-13 11:12:18 MSK, ends at Tue 2022-01-25 14:49:41 окт 13 17:02:49 hostname_fleet systemd[1]: Started Elastic Agent is a unified окт 13 17:02:50 hostname_fleet elastic-agent[8507]: 2021-10-13T17:02:50.143+0300 окт 13 17:02:50 hostname_fleet elastic-agent[8507]: 2021-10-13T17:02:50.144+0300 окт 13 17:02:50 hostname_fleet elastic-agent[8507]: 2021-10-13T17:02:50.144+0300 окт 13 17:02:50 hostname_fleet elastic-agent[8507]: 2021-10-13T17:02:50.502+0300 окт 13 17:02:50 hostname_fleet elastic-agent[8507]: 2021-10-13T17:02:50.503+0300 окт 13 17:02:50 hostname_fleet elastic-agent[8507]: 2021-10-13T17:02:50.503+0300 окт 13 17:02:50 hostname_fleet elastic-agent[8507]: 2021-10-13T17:02:50.504+0300 окт 13 17:02:50 hostname_fleet elastic-agent[8507]: 2021-10-13T17:02:50.519+0300 окт 13 17:02:50 hostname_fleet elastic-agent[8507]: 2021-10-13T17:02:50.520+0300 окт 13 17:02:50 hostname_fleet elastic-agent[8507]: 2021-10-13T17:02:50.527+0300 окт 13 17:02:50 hostname_fleet elastic-agent[8507]: 2021-10-13T17:02:50.527+0300 окт 13 17:03:01 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:01.880+0300 окт 13 17:03:03 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:03.327+0300 окт 13 17:03:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:17.053+0300 окт 13 17:03:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:17.556+0300 окт 13 17:03:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:17.752+0300 окт 13 17:03:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:17.754+0300 окт 13 17:03:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:17.754+0300 окт 13 17:03:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:17.758+0300 окт 13 17:03:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:17.758+0300 окт 13 17:03:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:17.759+0300 окт 13 17:03:18 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:18.450+0300 окт 13 17:03:19 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:19.064+0300 окт 13 17:03:27 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:27.755+0300 окт 13 17:03:37 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:37.756+0300 окт 13 17:03:47 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:47.757+0300 окт 13 17:03:57 hostname_fleet elastic-agent[8507]: 2021-10-13T17:03:57.758+0300 окт 13 17:04:07 hostname_fleet elastic-agent[8507]: 2021-10-13T17:04:07.759+0300 окт 13 17:04:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:04:17.760+0300 окт 13 17:04:27 hostname_fleet elastic-agent[8507]: 2021-10-13T17:04:27.762+0300 окт 13 17:04:37 hostname_fleet elastic-agent[8507]: 2021-10-13T17:04:37.763+0300 окт 13 17:04:47 hostname_fleet elastic-agent[8507]: 2021-10-13T17:04:47.763+0300 окт 13 17:04:57 hostname_fleet elastic-agent[8507]: 2021-10-13T17:04:57.764+0300 окт 13 17:05:07 hostname_fleet elastic-agent[8507]: 2021-10-13T17:05:07.764+0300 окт 13 17:05:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:05:17.766+0300 окт 13 17:05:27 hostname_fleet elastic-agent[8507]: 2021-10-13T17:05:27.766+0300 окт 13 17:05:37 hostname_fleet elastic-agent[8507]: 2021-10-13T17:05:37.768+0300 окт 13 17:05:47 hostname_fleet elastic-agent[8507]: 2021-10-13T17:05:47.768+0300 окт 13 17:05:57 hostname_fleet elastic-agent[8507]: 2021-10-13T17:05:57.769+0300 окт 13 17:06:07 hostname_fleet elastic-agent[8507]: 2021-10-13T17:06:07.770+0300 окт 13 17:06:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:06:17.770+0300 окт 13 17:06:27 hostname_fleet elastic-agent[8507]: 2021-10-13T17:06:27.771+0300 окт 13 17:06:37 hostname_fleet elastic-agent[8507]: 2021-10-13T17:06:37.772+0300 окт 13 17:06:47 hostname_fleet elastic-agent[8507]: 2021-10-13T17:06:47.772+0300 окт 13 17:06:57 hostname_fleet elastic-agent[8507]: 2021-10-13T17:06:57.773+0300 окт 13 17:07:07 hostname_fleet elastic-agent[8507]: 2021-10-13T17:07:07.774+0300 окт 13 17:07:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:07:17.776+0300 окт 13 17:07:27 hostname_fleet elastic-agent[8507]: 2021-10-13T17:07:27.777+0300 окт 13 17:07:37 hostname_fleet elastic-agent[8507]: 2021-10-13T17:07:37.778+0300 окт 13 17:07:47 hostname_fleet elastic-agent[8507]: 2021-10-13T17:07:47.779+0300 окт 13 17:07:57 hostname_fleet elastic-agent[8507]: 2021-10-13T17:07:57.779+0300 окт 13 17:08:07 hostname_fleet elastic-agent[8507]: 2021-10-13T17:08:07.780+0300 окт 13 17:08:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:08:17.781+0300 окт 13 17:08:27 hostname_fleet elastic-agent[8507]: 2021-10-13T17:08:27.782+0300 окт 13 17:08:37 hostname_fleet elastic-agent[8507]: 2021-10-13T17:08:37.783+0300 окт 13 17:08:47 hostname_fleet elastic-agent[8507]: 2021-10-13T17:08:47.783+0300 окт 13 17:08:57 hostname_fleet elastic-agent[8507]: 2021-10-13T17:08:57.784+0300 окт 13 17:09:07 hostname_fleet elastic-agent[8507]: 2021-10-13T17:09:07.785+0300 окт 13 17:09:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:09:17.785+0300 окт 13 17:09:27 hostname_fleet elastic-agent[8507]: 2021-10-13T17:09:27.786+0300 окт 13 17:09:37 hostname_fleet elastic-agent[8507]: 2021-10-13T17:09:37.787+0300 окт 13 17:09:47 hostname_fleet elastic-agent[8507]: 2021-10-13T17:09:47.788+0300 окт 13 17:09:57 hostname_fleet elastic-agent[8507]: 2021-10-13T17:09:57.789+0300 окт 13 17:10:07 hostname_fleet elastic-agent[8507]: 2021-10-13T17:10:07.790+0300 окт 13 17:10:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:10:17.791+0300 окт 13 17:10:27 hostname_fleet elastic-agent[8507]: 2021-10-13T17:10:27.792+0300 окт 13 17:10:37 hostname_fleet elastic-agent[8507]: 2021-10-13T17:10:37.792+0300 окт 13 17:10:47 hostname_fleet elastic-agent[8507]: 2021-10-13T17:10:47.794+0300 окт 13 17:10:57 hostname_fleet elastic-agent[8507]: 2021-10-13T17:10:57.795+0300 окт 13 17:11:07 hostname_fleet elastic-agent[8507]: 2021-10-13T17:11:07.796+0300 окт 13 17:11:17 hostname_fleet elastic-agent[8507]: 2021-10-13T17:11:17.796+0300 окт 13 17:11:27 hostname_fleet elastic-agent[8507]: 2021-10-13T17:11:27.797+0300 окт 13 17:11:37 hostname_fleet elastic-agent[8507]: 2021-10-13T17:11:37.798+0300 окт 13 17:12:49 hostname_fleet elastic-agent[8507]: 2021-10-13T17:12:49.421+0300 окт 13 17:12:49 hostname_fleet elastic-agent[8507]: 2021-10-13T17:12:49.421+0300 окт 13 17:12:49 hostname_fleet elastic-agent[8507]: 2021-10-13T17:12:49.421+0300 окт 13 17:12:49 hostname_fleet systemd[1]: Stopping Elastic Agent is a окт 13 17:12:50 hostname_fleet elastic-agent[8507]: 2021-10-13T17:12:50.923+0300 окт 13 17:12:50 hostname_fleet elastic-agent[8507]: 2021-10-13T17:12:50.923+0300 окт 13 17:13:20 hostname_fleet elastic-agent[8507]: 2021-10-13T17:13:20.601+0300 окт 13 17:13:20 hostname_fleet elastic-agent[8507]: 2021-10-13T17:13:20.601+0300 окт 13 17:13:30 hostname_fleet elastic-agent[8507]: 2021-10-13T17:13:30.966+0300 окт 13 17:13:30 hostname_fleet elastic-agent[8507]: 2021-10-13T17:13:30.966+0300 окт 13 17:14:11 hostname_fleet elastic-agent[8507]: 2021-10-13T17:14:11.009+0300 окт 13 17:14:11 hostname_fleet elastic-agent[8507]: 2021-10-13T17:14:11.009+0300 окт 13 17:14:11 hostname_fleet elastic-agent[8507]: 2021-10-13T17:14:11.009+0300 окт 13 17:14:11 hostname_fleet elastic-agent[8507]: 2021-10-13T17:14:11.009+0300 окт 13 17:14:11 hostname_fleet systemd[1]: elastic-agent.service: Killing окт 13 17:14:11 hostname_fleet systemd[1]: elastic-agent.service: Succeeded.
окт 13 17:14:11 hostname_fleet systemd[1]: Stopped Elastic Agent is a unified окт 13 17:14:11 hostname_fleet systemd[1]: elastic-agent.service: Consumed окт 13 17:14:18 hostname_fleet systemd[1]: Started Elastic Agent is a unified окт 13 17:14:18 hostname_fleet systemd[1]: Stopping Elastic Agent is a окт 13 17:14:18 hostname_fleet elastic-agent[8746]: 2021-10-13T17:14:18.796+0300 окт 13 17:14:18 hostname_fleet elastic-agent[8746]: 2021-10-13T17:14:18.797+0300 окт 13 17:14:18 hostname_fleet elastic-agent[8746]: 2021-10-13T17:14:18.797+0300 окт 13 17:14:19 hostname_fleet elastic-agent[8746]: 2021-10-13T17:14:19.294+0300 окт 13 17:14:19 hostname_fleet elastic-agent[8746]: 2021-10-13T17:14:19.295+0300 окт 13 17:14:19 hostname_fleet elastic-agent[8746]: 2021-10-13T17:14:19.295+0300 окт 13 17:14:19 hostname_fleet elastic-agent[8746]: 2021-10-13T17:14:19.295+0300 окт 13 17:14:19 hostname_fleet elastic-agent[8746]: 2021-10-13T17:14:19.295+0300 окт 13 17:14:19 hostname_fleet elastic-agent[8746]: 2021-10-13T17:14:19.296+0300 окт 13 17:14:19 hostname_fleet elastic-agent[8746]: 2021-10-13T17:14:19.297+0300 окт 13 17:14:19 hostname_fleet elastic-agent[8746]: 2021-10-13T17:14:19.297+0300 окт 13 17:14:19 hostname_fleet elastic-agent[8746]: 2021-10-13T17:14:19.297+0300 окт 13 17:14:19 hostname_fleet systemd[1]: elastic-agent.service: Succeeded.
окт 13 17:14:19 hostname_fleet systemd[1]: Stopped Elastic Agent is a unified MSK. --
agent to observe, monitor and protect your system..
INFO application/application.go:67 Detecting execution mode
INFO application/application.go:76 Agent is managed locally
INFO capabilities/capabilities.go:59 capabilities file not found in /opt/Elastic/Agent/capabilities.yml
INFO [composable.providers.docker] docker/docker.go:43 Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
INFO [api] api/server.go:62 Starting stats endpoint
INFO application/local_mode.go:168 Agent is starting
INFO [api] api/server.go:64 Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
INFO application/local_mode.go:178 Agent is stopped
INFO application/periodic.go:79 Configuration changes detected
INFO stateresolver/stateresolver.go:48 New State ID is re82v5az
INFO stateresolver/stateresolver.go:49 Converging state requires execution of 2 step(s)
INFO log/reporter.go:40 2021-10-13T17:03:01+03:00 - message: Application: metricbeat--7.15.0[6dc6e489-5a9e-4305-813d-860a820d08e8]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
INFO log/reporter.go:40 2021-10-13T17:03:03+03:00 - message: Application: metricbeat--7.15.0[6dc6e489-5a9e-4305-813d-860a820d08e8]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
INFO log/reporter.go:40 2021-10-13T17:03:17+03:00 - message: Application: filebeat--7.15.0--36643631373035623733363936343635[6dc6e489-5a9e-4305-813d-860a820d08e8]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
INFO operation/operator.go:269 operation 'operation-install' skipped for metricbeat.7.15.0
INFO log/reporter.go:40 2021-10-13T17:03:17+03:00 - message: Application: metricbeat--7.15.0--36643631373035623733363936343635[6dc6e489-5a9e-4305-813d-860a820d08e8]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
INFO stateresolver/stateresolver.go:66 Updating internal state
INFO application/periodic.go:101 No configuration change
INFO stateresolver/stateresolver.go:48 New State ID is re82v5az
INFO stateresolver/stateresolver.go:49 Converging state requires execution of 0 step(s)
INFO stateresolver/stateresolver.go:66 Updating internal state
INFO log/reporter.go:40 2021-10-13T17:03:18+03:00 - message: Application: filebeat--7.15.0--36643631373035623733363936343635[6dc6e489-5a9e-4305-813d-860a820d08e8]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
INFO log/reporter.go:40 2021-10-13T17:03:19+03:00 - message: Application: metricbeat--7.15.0--36643631373035623733363936343635[6dc6e489-5a9e-4305-813d-860a820d08e8]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO application/periodic.go:101 No configuration change
INFO cmd/run.go:182 Shutting down Elastic Agent and sending last events...
INFO operation/operator.go:201 waiting for installer of pipeline 'default' to finish
INFO process/app.go:176 Signaling application to stop because of shutdown: metricbeat--7.15.0
unified agent to observe, monitor and protect your system....
INFO process/app.go:176 Signaling application to stop because of shutdown: filebeat--7.15.0--36643631373035623733363936343635
INFO log/reporter.go:40 2021-10-13T17:12:50+03:00 - message: Application: metricbeat--7.15.0[6dc6e489-5a9e-4305-813d-860a820d08e8]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
WARN status/reporter.go:236 Elastic Agent status changed to: 'degraded'
INFO log/reporter.go:40 2021-10-13T17:13:20+03:00 - message: Application: metricbeat--7.15.0--36643631373035623733363936343635[6dc6e489-5a9e-4305-813d-860a820d08e8]: State changed to DEGRADED: Missed last check-in - type: 'STATE' - sub_type: 'RUNNING'
INFO process/app.go:176 Signaling application to stop because of shutdown: metricbeat--7.15.0--36643631373035623733363936343635
INFO log/reporter.go:40 2021-10-13T17:13:30+03:00 - message: Application: filebeat--7.15.0--36643631373035623733363936343635[6dc6e489-5a9e-4305-813d-860a820d08e8]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
INFO status/reporter.go:236 Elastic Agent status changed to: 'online'
INFO log/reporter.go:40 2021-10-13T17:14:11+03:00 - message: Application: metricbeat--7.15.0--36643631373035623733363936343635[6dc6e489-5a9e-4305-813d-860a820d08e8]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
INFO cmd/run.go:190 Shutting down completed.
INFO [api] api/server.go:66 Stats endpoint (/opt/Elastic/Agent/data/tmp/elastic-agent.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/elastic-agent.sock: use of closed network connection
process 8629 (metricbeat) with signal SIGKILL.
agent to observe, monitor and protect your system..
12.934s CPU time.
agent to observe, monitor and protect your system..
unified agent to observe, monitor and protect your system....
INFO application/application.go:67 Detecting execution mode
INFO application/application.go:76 Agent is managed locally
INFO capabilities/capabilities.go:59 capabilities file not found in /opt/Elastic/Agent/capabilities.yml
INFO [composable.providers.docker] docker/docker.go:43 Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
INFO [api] api/server.go:62 Starting stats endpoint
INFO application/local_mode.go:168 Agent is starting
INFO [api] api/server.go:64 Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
INFO application/local_mode.go:178 Agent is stopped
INFO application/periodic.go:79 Configuration changes detected
INFO cmd/run.go:182 Shutting down Elastic Agent and sending last events...
INFO cmd/run.go:190 Shutting down completed.
INFO [api] api/server.go:66 Stats endpoint (/opt/Elastic/Agent/data/tmp/elastic-agent.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/elastic-agent.sock: use of closed network connection
agent to observe, monitor and protect your system..

b.kaliuzhnyi · January 25, 2022, 12:18pm

окт 14 10:44:55 hostname_fleet systemd[1]: Started Elastic Agent is a unified agent to observe, monitor and protect your system..
окт 14 10:44:55 hostname_fleet elastic-agent[12330]: 2021-10-14T10:44:55.318+0300 INFO application/application.go:67 Detecting execution mode
окт 14 10:44:55 hostname_fleet elastic-agent[12330]: 2021-10-14T10:44:55.319+0300 INFO application/application.go:76 Agent is managed locally
окт 14 10:44:55 hostname_fleet elastic-agent[12330]: 2021-10-14T10:44:55.319+0300 INFO capabilities/capabilities.go:59 capabilities file not found in /opt/Elastic/Agent/capabilities.yml
окт 14 10:44:55 hostname_fleet elastic-agent[12330]: 2021-10-14T10:44:55.818+0300 INFO [composable.providers.docker] docker/docker.go:43 Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
окт 14 10:44:55 hostname_fleet elastic-agent[12330]: 2021-10-14T10:44:55.819+0300 INFO [api] api/server.go:62 Starting stats endpoint
окт 14 10:44:55 hostname_fleet elastic-agent[12330]: 2021-10-14T10:44:55.819+0300 INFO application/local_mode.go:168 Agent is starting
окт 14 10:44:55 hostname_fleet elastic-agent[12330]: 2021-10-14T10:44:55.819+0300 INFO [api] api/server.go:64 Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
окт 14 10:44:55 hostname_fleet elastic-agent[12330]: 2021-10-14T10:44:55.820+0300 INFO application/local_mode.go:178 Agent is stopped
окт 14 10:44:55 hostname_fleet elastic-agent[12330]: 2021-10-14T10:44:55.821+0300 INFO application/periodic.go:79 Configuration changes detected
окт 14 10:44:55 hostname_fleet elastic-agent[12330]: 2021-10-14T10:44:55.829+0300 INFO stateresolver/stateresolver.go:48 New State ID is re82v5az
окт 14 10:44:55 hostname_fleet elastic-agent[12330]: 2021-10-14T10:44:55.829+0300 INFO stateresolver/stateresolver.go:49 Converging state requires execution of 2 step(s)
окт 14 10:45:06 hostname_fleet elastic-agent[12330]: 2021-10-14T10:45:06.574+0300 INFO log/reporter.go:40 2021-10-14T10:45:06+03:00 - message: Application: metricbeat--7.15.0[0bc62b36-1e5a-498e-8763-764a95ab6aee]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
окт 14 10:45:08 hostname_fleet elastic-agent[12330]: 2021-10-14T10:45:08.386+0300 INFO log/reporter.go:40 2021-10-14T10:45:08+03:00 - message: Application: metricbeat--7.15.0[0bc62b36-1e5a-498e-8763-764a95ab6aee]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
окт 14 10:45:19 hostname_fleet elastic-agent[12330]: 2021-10-14T10:45:19.475+0300 INFO log/reporter.go:40 2021-10-14T10:45:19+03:00 - message: Application: filebeat--7.15.0--36643631373035623733363936343635[0bc62b36-1e5a-498e-8763-764a95ab6aee]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
окт 14 10:45:19 hostname_fleet elastic-agent[12330]: 2021-10-14T10:45:19.912+0300 INFO operation/operator.go:269 operation 'operation-install' skipped for metricbeat.7.15.0
окт 14 10:45:20 hostname_fleet elastic-agent[12330]: 2021-10-14T10:45:20.513+0300 INFO log/reporter.go:40 2021-10-14T10:45:20+03:00 - message: Application: metricbeat--7.15.0--36643631373035623733363936343635[0bc62b36-1e5a-498e-8763-764a95ab6aee]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
окт 14 10:45:20 hostname_fleet elastic-agent[12330]: 2021-10-14T10:45:20.516+0300 INFO stateresolver/stateresolver.go:66 Updating internal state
окт 14 10:45:20 hostname_fleet elastic-agent[12330]: 2021-10-14T10:45:20.517+0300 INFO application/periodic.go:101 No configuration change
окт 14 10:45:20 hostname_fleet elastic-agent[12330]: 2021-10-14T10:45:20.520+0300 INFO stateresolver/stateresolver.go:48 New State ID is re82v5az
окт 14 10:45:20 hostname_fleet elastic-agent[12330]: 2021-10-14T10:45:20.520+0300 INFO stateresolver/stateresolver.go:49 Converging state requires execution of 0 step(s)
окт 14 10:45:20 hostname_fleet elastic-agent[12330]: 2021-10-14T10:45:20.520+0300 INFO stateresolver/stateresolver.go:66 Updating internal state
окт 14 10:45:20 hostname_fleet elastic-agent[12330]: 2021-10-14T10:45:20.792+0300 INFO log/reporter.go:40 2021-10-14T10:45:20+03:00 - message: Application: filebeat--7.15.0--36643631373035623733363936343635[0bc62b36-1e5a-498e-8763-764a95ab6aee]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
окт 14 10:45:21 hostname_fleet elastic-agent[12330]: 2021-10-14T10:45:21.791+0300 INFO log/reporter.go:40 2021-10-14T10:45:21+03:00 - message: Application: metricbeat--7.15.0--36643631373035623733363936343635[0bc62b36-1e5a-498e-8763-764a95ab6aee]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
окт 14 10:45:30 hostname_fleet elastic-agent[12330]: 2021-10-14T10:45:30.517+0300 INFO application/periodic.go:101 No configuration change
окт 14 10:46:43 hostname_fleet elastic-agent[12330]: 2021-10-14T10:46:43.303+0300 INFO cmd/run.go:182 Shutting down Elastic Agent and sending last events...
окт 14 10:46:43 hostname_fleet elastic-agent[12330]: 2021-10-14T10:46:43.303+0300 INFO operation/operator.go:201 waiting for installer of pipeline 'default' to finish
окт 14 10:46:43 hostname_fleet elastic-agent[12330]: 2021-10-14T10:46:43.303+0300 INFO process/app.go:176 Signaling application to stop because of shutdown: metricbeat--7.15.0--36643631373035623733363936343635
окт 14 10:46:43 hostname_fleet systemd[1]: Stopping Elastic Agent is a unified agent to observe, monitor and protect your system....
окт 14 10:46:44 hostname_fleet elastic-agent[12330]: 2021-10-14T10:46:44.804+0300 INFO process/app.go:176 Signaling application to stop because of shutdown: metricbeat--7.15.0
окт 14 10:46:44 hostname_fleet elastic-agent[12330]: 2021-10-14T10:46:44.804+0300 INFO log/reporter.go:40 2021-10-14T10:46:44+03:00 - message: Application: metricbeat--7.15.0--36643631373035623733363936343635[0bc62b36-1e5a-498e-8763-764a95ab6aee]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
окт 14 10:47:24 hostname_fleet elastic-agent[12330]: 2021-10-14T10:47:24.845+0300 INFO process/app.go:176 Signaling application to stop because of shutdown: filebeat--7.15.0--36643631373035623733363936343635
окт 14 10:47:24 hostname_fleet elastic-agent[12330]: 2021-10-14T10:47:24.845+0300 INFO log/reporter.go:40 2021-10-14T10:47:24+03:00 - message: Application: metricbeat--7.15.0[0bc62b36-1e5a-498e-8763-764a95ab6aee]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
окт 14 10:47:25 hostname_fleet elastic-agent[12330]: 2021-10-14T10:47:25.844+0300 WARN status/reporter.go:236 Elastic Agent status changed to: 'degraded'
окт 14 10:47:25 hostname_fleet elastic-agent[12330]: 2021-10-14T10:47:25.844+0300 INFO log/reporter.go:40 2021-10-14T10:47:25+03:00 - message: Application: filebeat--7.15.0--36643631373035623733363936343635[0bc62b36-1e5a-498e-8763-764a95ab6aee]: State changed to DEGRADED: Missed last check-in - type: 'STATE' - sub_type: 'RUNNING'
окт 14 10:48:04 hostname_fleet elastic-agent[12330]: 2021-10-14T10:48:04.880+0300 INFO status/reporter.go:236 Elastic Agent status changed to: 'online'
окт 14 10:48:04 hostname_fleet elastic-agent[12330]: 2021-10-14T10:48:04.880+0300 INFO log/reporter.go:40 2021-10-14T10:48:04+03:00 - message: Application: filebeat--7.15.0--36643631373035623733363936343635[0bc62b36-1e5a-498e-8763-764a95ab6aee]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
окт 14 10:48:04 hostname_fleet elastic-agent[12330]: 2021-10-14T10:48:04.881+0300 INFO cmd/run.go:190 Shutting down completed.
окт 14 10:48:04 hostname_fleet elastic-agent[12330]: 2021-10-14T10:48:04.881+0300 INFO [api] api/server.go:66 Stats endpoint (/opt/Elastic/Agent/data/tmp/elastic-agent.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/elastic-agent.sock: use of closed network connection
окт 14 10:48:04 hostname_fleet systemd[1]: elastic-agent.service: Killing process 12367 (filebeat) with signal SIGKILL.
окт 14 10:48:04 hostname_fleet systemd[1]: elastic-agent.service: Succeeded.
окт 14 10:48:04 hostname_fleet systemd[1]: Stopped Elastic Agent is a unified agent to observe, monitor and protect your system..
окт 14 10:48:04 hostname_fleet systemd[1]: elastic-agent.service: Consumed 9.942s CPU time.
окт 14 10:48:11 hostname_fleet systemd[1]: Started Elastic Agent is a unified agent to observe, monitor and protect your system..
окт 14 10:48:11 hostname_fleet systemd[1]: Stopping Elastic Agent is a unified agent to observe, monitor and protect your system....
окт 14 10:48:11 hostname_fleet systemd[1]: elastic-agent.service: Succeeded.
окт 14 10:48:11 hostname_fleet systemd[1]: Stopped Elastic Agent is a unified agent to observe, monitor and protect your system..
окт 14 10:58:08 hostname_fleet systemd[1]: Started Agent manages other beats based on configuration provided..
окт 14 10:58:08 hostname_fleet systemd[12712]: elastic-agent.service: Failed to locate executable /usr/bin/elastic-agent: No such file or directory
окт 14 10:58:08 hostname_fleet systemd[12712]: elastic-agent.service: Failed at step EXEC spawning /usr/bin/elastic-agent: No such file or directory
окт 14 10:58:08 hostname_fleet systemd[1]: elastic-agent.service: Main process exited, code=exited, status=203/EXEC
окт 14 10:58:08 hostname_fleet systemd[1]: elastic-agent.service: Failed with result 'exit-code'.
окт 14 10:58:08 hostname_fleet systemd[1]: elastic-agent.service: Scheduled restart job, restart counter is at 1.
окт 14 10:58:08 hostname_fleet systemd[1]: Stopped Agent manages other beats based on configuration provided..
окт 14 10:58:08 hostname_fleet systemd[1]: Started Agent manages other beats based on configuration provided..
окт 14 10:58:08 hostname_fleet systemd[12713]: elastic-agent.service: Failed to locate executable /usr/bin/elastic-agent: No such file or directory
окт 14 10:58:08 hostname_fleet systemd[12713]: elastic-agent.service: Failed at step EXEC spawning /usr/bin/elastic-agent: No such file or directory
окт 14 10:58:08 hostname_fleet systemd[1]: elastic-agent.service: Main process exited, code=exited, status=203/EXEC
окт 14 10:58:08 hostname_fleet systemd[1]: elastic-agent.service: Failed with result 'exit-code'.
окт 14 10:58:08 hostname_fleet systemd[1]: elastic-agent.service: Scheduled restart job, restart counter is at 2.
окт 14 10:58:08 hostname_fleet systemd[1]: Stopped Agent manages other beats based on configuration provided..
окт 14 10:58:08 hostname_fleet systemd[1]: Started Agent manages other beats based on configuration provided..
окт 14 10:58:08 hostname_fleet systemd[12714]: elastic-agent.service: Failed to locate executable /usr/bin/elastic-agent: No such file or directory
окт 14 10:58:08 hostname_fleet systemd[12714]: elastic-agent.service: Failed at step EXEC spawning /usr/bin/elastic-agent: No such file or directory
окт 14 10:58:08 hostname_fleet systemd[1]: elastic-agent.service: Main process exited, code=exited, status=203/EXEC
окт 14 10:58:08 hostname_fleet systemd[1]: elastic-agent.service: Failed with result 'exit-code'.
окт 14 10:58:09 hostname_fleet systemd[1]: elastic-agent.service: Scheduled restart job, restart counter is at 3.
окт 14 10:58:09 hostname_fleet systemd[1]: Stopped Agent manages other beats based on configuration provided..
окт 14 10:58:09 hostname_fleet systemd[1]: Started Agent manages other beats based on configuration provided..
окт 14 10:58:09 hostname_fleet systemd[12715]: elastic-agent.service: Failed to locate executable /usr/bin/elastic-agent: No such file or directory
окт 14 10:58:09 hostname_fleet systemd[12715]: elastic-agent.service: Failed at step EXEC spawning /usr/bin/elastic-agent: No such file or directory
окт 14 10:58:09 hostname_fleet systemd[1]: elastic-agent.service: Main process exited, code=exited, status=203/EXEC
окт 14 10:58:09 hostname_fleet systemd[1]: elastic-agent.service: Failed with result 'exit-code'.
окт 14 10:58:09 hostname_fleet systemd[1]: elastic-agent.service: Scheduled restart job, restart counter is at 4.
окт 14 10:58:09 hostname_fleet systemd[1]: Stopped Agent manages other beats based on configuration provided..
окт 14 10:58:09 hostname_fleet systemd[1]: Started Agent manages other beats based on configuration provided..
окт 14 10:58:09 hostname_fleet systemd[12716]: elastic-agent.service: Failed to locate executable /usr/bin/elastic-agent: No such file or directory
окт 14 10:58:09 hostname_fleet systemd[12716]: elastic-agent.service: Failed at step EXEC spawning /usr/bin/elastic-agent: No such file or directory
окт 14 10:58:09 hostname_fleet systemd[1]: elastic-agent.service: Main process exited, code=exited, status=203/EXEC
окт 14 10:58:09 hostname_fleet systemd[1]: elastic-agent.service: Failed with result 'exit-code'.
окт 14 10:58:09 hostname_fleet systemd[1]: elastic-agent.service: Scheduled restart job, restart counter is at 5.
окт 14 10:58:09 hostname_fleet systemd[1]: Stopped Agent manages other beats based on configuration provided..
окт 14 10:58:09 hostname_fleet systemd[1]: elastic-agent.service: Start request repeated too quickly.
окт 14 10:58:09 hostname_fleet systemd[1]: elastic-agent.service: Failed with result 'exit-code'.
окт 14 10:58:09 hostname_fleet systemd[1]: Failed to start Agent manages other beats based on configuration provided..
окт 14 11:04:01 hostname_fleet systemd[1]: Started Agent manages other beats based on configuration provided..
окт 14 11:04:01 hostname_fleet systemd[1]: elastic-agent.service: Main process exited, code=exited, status=203/EXEC
окт 14 11:04:01 hostname_fleet systemd[1]: elastic-agent.service: Failed with result 'exit-code'.
окт 14 11:04:01 hostname_fleet systemd[1]: elastic-agent.service: Scheduled restart job, restart counter is at 1.
окт 14 11:04:01 hostname_fleet systemd[1]: Stopped Agent manages other beats based on configuration provided..
окт 14 11:04:01 hostname_fleet systemd[12841]: elastic-agent.service: Failed to locate executable /usr/bin/elastic-agent: No such file or directory
окт 14 11:04:01 hostname_fleet systemd[12841]: elastic-agent.service: Failed at step EXEC spawning /usr/bin/elastic-agent: No such file or directory
окт 14 11:04:01 hostname_fleet systemd[1]: Started Agent manages other beats based on configuration provided..
окт 14 11:04:01 hostname_fleet systemd[1]: elastic-agent.service: Main process exited, code=exited, status=203/EXEC
окт 14 11:04:01 hostname_fleet systemd[1]: elastic-agent.service: Failed with result 'exit-code'.
окт 14 11:04:01 hostname_fleet systemd[1]: elastic-agent.service: Scheduled restart job, restart counter is at 2.
окт 14 11:04:01 hostname_fleet systemd[1]: Stopped Agent manages other beats based on configuration provided..
окт 14 11:04:01 hostname_fleet systemd[1]: Started Agent manages other beats based on configuration provided..
окт 14 11:04:01 hostname_fleet systemd[12843]: elastic-agent.service: Failed to locate executable /usr/bin/elastic-agent: No such file or directory
окт 14 11:04:01 hostname_fleet systemd[12843]: elastic-agent.service: Failed at step EXEC spawning /usr/bin/elastic-agent: No such file or directory
окт 14 11:04:01 hostname_fleet systemd[1]: elastic-agent.service: Main process exited, code=exited, status=203/EXEC
окт 14 11:04:01 hostname_fleet systemd[1]: elastic-agent.service: Failed with result 'exit-code'.
окт 14 11:04:02 hostname_fleet systemd[1]: elastic-agent.service: Scheduled restart job, restart counter is at 3.
окт 14 11:04:02 hostname_fleet systemd[1]: Stopped Agent manages other beats based on configuration provided..
окт 14 11:04:02 hostname_fleet systemd[1]: Started Agent manages other beats based on configuration provided..
окт 14 11:04:02 hostname_fleet systemd[12845]: elastic-agent.service: Failed to locate executable /usr/bin/elastic-agent: No such file or directory
окт 14 11:04:02 hostname_fleet systemd[12845]: elastic-agent.service: Failed at step EXEC spawning /usr/bin/elastic-agent: No such file or directory
окт 14 11:04:02 hostname_fleet systemd[1]: elastic-agent.service: Main process exited, code=exited, status=203/EXEC
окт 14 11:04:02 hostname_fleet systemd[1]: elastic-agent.service: Failed with result 'exit-code'.
окт 14 11:04:02 hostname_fleet systemd[1]: elastic-agent.service: Scheduled restart job, restart counter is at 4.
окт 14 11:04:02 hostname_fleet systemd[1]: Stopped Agent manages other beats based on configuration provided..
окт 14 11:04:02 hostname_fleet systemd[1]: Started Agent manages other beats based on configuration provided..
окт 14 11:04:02 hostname_fleet systemd[12846]: elastic-agent.service: Failed to locate executable /usr/bin/elastic-agent: No such file or directory
окт 14 11:04:02 hostname_fleet systemd[12846]: elastic-agent.service: Failed at step EXEC spawning /usr/bin/elastic-agent: No such file or directory
окт 14 11:04:02 hostname_fleet systemd[1]: elastic-agent.service: Main process exited, code=exited, status=203/EXEC
окт 14 11:04:02 hostname_fleet systemd[1]: elastic-agent.service: Failed with result 'exit-code'.
окт 14 11:04:02 hostname_fleet systemd[1]: elastic-agent.service: Scheduled restart job, restart counter is at 5.
окт 14 11:04:02 hostname_fleet systemd[1]: Stopped Agent manages other beats based on configuration provided..
окт 14 11:04:02 hostname_fleet systemd[1]: elastic-agent.service: Start request repeated too quickly.
окт 14 11:04:02 hostname_fleet systemd[1]: elastic-agent.service: Failed with result 'exit-code'.
окт 14 11:04:02 hostname_fleet systemd[1]: Failed to start Agent manages other beats based on configuration provided..
-- Boot 9174faa7e4104287b704d6854509eb68 --
янв 21 16:19:29 hostname_fleet systemd[1]: Started Agent manages other beats based on configuration provided..
янв 21 16:20:24 hostname_fleet systemd[1]: Stopping Agent manages other beats based on configuration provided....
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: State 'stop-sigterm' timed out. Killing.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414183 (elastic-agent) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414220 (filebeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414241 (metricbeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414191 (elastic-agent) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414199 (elastic-agent) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414221 (elastic-agent) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414222 (elastic-agent) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414223 (filebeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414224 (filebeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414225 (filebeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414226 (filebeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414227 (filebeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414228 (filebeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414229 (filebeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414242 (metricbeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414243 (metricbeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414244 (metricbeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414245 (metricbeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414246 (metricbeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414247 (metricbeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Killing process 414248 (metricbeat) with signal SIGKILL.
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Main process exited, code=killed, status=9/KILL
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Failed with result 'timeout'.
янв 21 16:21:54 hostname_fleet systemd[1]: Stopped Agent manages other beats based on configuration provided..
янв 21 16:21:54 hostname_fleet systemd[1]: elastic-agent.service: Consumed 7.235s CPU time.
янв 21 16:21:54 hostname_fleet systemd[1]: Started Agent manages other beats based on configuration provided..

mtojek · January 26, 2022, 2:53pm

Hi,

did you try executing elastic-agent status with sudo?

b.kaliuzhnyi · January 26, 2022, 3:02pm

I work as root.

The issue was resolved by reinstalling the fleet server with the correct Elastic CA certificate and specifying the certificate in text in the Elasticsearch output configuration.

I don't know which of these helped, but the events are coming normally))

system · February 23, 2022, 5:03pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.