Fail to checkin to fleet server

subash · November 15, 2022, 7:36am

We are using Elasticsearch 7.17.4 on private cloud.

We have 4 elastic agents installed on windows 10 PCs and managed by fleet. It is observed that the Elastic agent status switches between 'Healthy' and 'Offline'.

Below is the error log
Kindly help to get this issue resolved.
Thanks in advance.

[elastic_agent][error] Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post "https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?": EOF

Screenshot

warkolm · November 15, 2022, 7:38am

Sharing more of the log would be helpful in providing context.

subash · November 15, 2022, 7:42am

Hi @warkolm ,

This is the only error message repeating again and again. I do not see any logs other than this.

Krishna_Teja · November 15, 2022, 12:32pm

Hi Subash

Can you share the details of the OS? Is it a Mac?

Regards
Krishna

subash · November 15, 2022, 3:04pm

Hi Krishna,

The agents are installed on Windows 10 PCs.

The integrations are System, Endpoint Security, OSquery_manager and Elastic_agent.

Regards
Subash

Krishna_Teja · November 15, 2022, 4:15pm

I've faced a similar issue with MacOS. Support team suggested to check network proxy, firewall settings. Everything was good but still the issue continued.

It seems the memory usage on server was high that caused outage sometimes. Can you check the resource usage.

Regards
Krishna

subash · November 18, 2022, 7:36am

Hi @warkolm,

I managed to get more logs from elastic agent. Please have a check if this helps.

{"log.level":"info","@timestamp":"2022-11-09T13:05:10.976Z","log.origin":{"file.name":"application/application.go","file.line":67},"message":"Detecting execution mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:10.978Z","log.origin":{"file.name":"application/application.go","file.line":92},"message":"Agent is managed by Fleet","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:10.978Z","log.origin":{"file.name":"capabilities/capabilities.go","file.line":59},"message":"capabilities file not found in C:\\Program Files\\Elastic\\Agent\\capabilities.yml","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-11-09T13:05:10.978Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":101},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:11.226Z","log.logger":"composable.providers.docker","log.origin":{"file.name":"docker/docker.go","file.line":43},"message":"Docker provider skipped, unable to connect: protocol not available","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:11.230Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":62},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:11.230Z","log.origin":{"file.name":"application/managed_mode.go","file.line":291},"message":"Agent is starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:11.230Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":64},"message":"Metrics endpoint listening on: \\\\.\\pipe\\elastic-agent (configured: npipe:///elastic-agent)","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-11-09T13:05:11.916Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":101},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-11-09T13:05:13.017Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":101},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-11-09T13:05:13.017Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":101},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:13.075Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is j5hv9vph","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:13.075Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 5 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:13.204Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for metricbeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:13.345Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-09T13:05:13Z - message: Application: metricbeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:15.818Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-09T13:05:15Z - message: Application: osquerybeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:19.567Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-09T13:05:19Z - message: Application: metricbeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:20.392Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-09T13:05:20Z - message: Application: endpoint-security--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:21.760Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-09T13:05:21Z - message: Application: osquerybeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:26.505Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-09T13:05:26Z - message: Application: filebeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:27.359Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for filebeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:27.461Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-09T13:05:27Z - message: Application: filebeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:28.069Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for metricbeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:28.194Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-09T13:05:28Z - message: Application: metricbeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:28.682Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:31.873Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-09T13:05:31Z - message: Application: filebeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:32.526Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-09T13:05:32Z - message: Application: filebeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:05:34.107Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-09T13:05:34Z - message: Application: metricbeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-09T13:05:51.053Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:06:05.642Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-09T13:06:05Z - message: Application: endpoint-security--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to CONFIG:  - type: 'STATE' - sub_type: 'CONFIG'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-09T13:06:05.742Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-09T13:06:05Z - message: Application: endpoint-security--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Protecting with policy {e079907c-d697-4d87-9f76-a5d5e48baacc} - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-09T13:07:27.220Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-09T13:10:03.289Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-09T13:16:30.985Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-10T07:59:44.641Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:05.439Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is 33G9TfKp","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:05.439Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 5 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:05.559Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for metricbeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:05.559Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-start' skipped for metricbeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:05.561Z","log.origin":{"file.name":"process/configure.go","file.line":50},"message":"initiating restart of 'metricbeat' due to config change","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:06.087Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:06Z - message: Application: metricbeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:06.263Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:06Z - message: Application: metricbeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:07.924Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for osquerybeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:07.924Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-start' skipped for osquerybeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:07.924Z","log.origin":{"file.name":"process/configure.go","file.line":50},"message":"initiating restart of 'osquerybeat' due to config change","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:08.449Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:08Z - message: Application: osquerybeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:08.621Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:08Z - message: Application: osquerybeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:12.099Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for endpoint-security.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:12.099Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-start' skipped for endpoint-security.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:12.223Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for filebeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:12.223Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-start' skipped for filebeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:12.224Z","log.origin":{"file.name":"process/configure.go","file.line":50},"message":"initiating restart of 'filebeat' due to config change","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:12.365Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:12Z - message: Application: metricbeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:12.755Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:12Z - message: Application: filebeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:13.091Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:13Z - message: Application: filebeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:13.744Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for filebeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:13.744Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-start' skipped for filebeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:13.746Z","log.origin":{"file.name":"process/configure.go","file.line":50},"message":"initiating restart of 'filebeat_monitoring' due to config change","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:14.273Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:14Z - message: Application: filebeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:14.434Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:14Z - message: Application: filebeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:14.683Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:14Z - message: Application: osquerybeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:15.063Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for metricbeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:15.063Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-start' skipped for metricbeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:15.065Z","log.origin":{"file.name":"process/configure.go","file.line":50},"message":"initiating restart of 'metricbeat_monitoring' due to config change","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:16.588Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:16Z - message: Application: metricbeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:16.651Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:16Z - message: Application: metricbeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:17.302Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:18.325Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:18Z - message: Application: filebeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:19.934Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:19Z - message: Application: filebeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T08:05:22.891Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T08:05:22Z - message: Application: metricbeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-10T08:05:39.450Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-10T08:07:09.037Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-10T08:09:35.821Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-10T08:17:01.497Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-10T08:24:09.812Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-10T14:57:48.689Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:17.625Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is gx8LiZLb","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:17.625Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 5 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:17.743Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for metricbeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:17.743Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-start' skipped for metricbeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:17.745Z","log.origin":{"file.name":"process/configure.go","file.line":50},"message":"initiating restart of 'metricbeat' due to config change","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:19.269Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:19Z - message: Application: metricbeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:19.386Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:19Z - message: Application: metricbeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:21.052Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for osquerybeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:21.052Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-start' skipped for osquerybeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:21.053Z","log.origin":{"file.name":"process/configure.go","file.line":50},"message":"initiating restart of 'osquerybeat' due to config change","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:21.578Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:21Z - message: Application: osquerybeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:21.707Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:21Z - message: Application: osquerybeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:24.580Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for endpoint-security.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:24.580Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-start' skipped for endpoint-security.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:24.707Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for filebeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:24.707Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-start' skipped for filebeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:24.708Z","log.origin":{"file.name":"process/configure.go","file.line":50},"message":"initiating restart of 'filebeat' due to config change","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:25.240Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:25Z - message: Application: filebeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:25.378Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:25Z - message: Application: filebeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:25.382Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:25Z - message: Application: metricbeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:26.004Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for filebeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:26.004Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-start' skipped for filebeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:26.009Z","log.origin":{"file.name":"process/configure.go","file.line":50},"message":"initiating restart of 'filebeat_monitoring' due to config change","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:27.533Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:27Z - message: Application: filebeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:27.620Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:27Z - message: Application: osquerybeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:27.689Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:27Z - message: Application: filebeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:28.319Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-install' skipped for metricbeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:28.319Z","log.origin":{"file.name":"operation/operator.go","file.line":284},"message":"operation 'operation-start' skipped for metricbeat.7.17.4","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:28.325Z","log.origin":{"file.name":"process/configure.go","file.line":50},"message":"initiating restart of 'metricbeat_monitoring' due to config change","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:28.645Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:28Z - message: Application: endpoint-security--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to CONFIG: Protecting with policy {e079907c-d697-4d87-9f76-a5d5e48baacc} - type: 'STATE' - sub_type: 'CONFIG'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:28.781Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:28Z - message: Application: endpoint-security--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Protecting with policy {e079907c-d697-4d87-9f76-a5d5e48baacc} - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:28.849Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:28Z - message: Application: metricbeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:28.944Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:28Z - message: Application: metricbeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:29.712Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:30.533Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:30Z - message: Application: filebeat--7.17.4[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:33.401Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:33Z - message: Application: filebeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-10T15:50:35.227Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-11-10T15:50:35Z - message: Application: metricbeat--7.17.4--36643631373035623733363936343635[3e35c734-8f13-4d85-8102-e9e8a33f0595]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-10T15:50:51.861Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-10T15:53:08.203Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-10T15:55:32.784Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-10T16:01:59.810Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-11-10T16:03:26.299Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://XXXXXXX:9243/api/fleet/agents/xxxxx-yyyy-zzzz-aaaa-bbbbbbbb/checkin?\": EOF","ecs.version":"1.6.0"}

subash · November 21, 2022, 1:42pm

Hi @ruflin,

Could you please help me on this?

Regards
Subash

system · December 19, 2022, 1:42pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Fail to checkin to fleet-server Elastic Agent fleet	17	5992	July 10, 2023
Unable to communicate with Fleet Server after Upgrade to 7.14 Beats fleet	16	6608	September 17, 2021
Elastic Agent / Endpoint no longer can connect to fleet server Elastic Agent	2	907	November 14, 2022
Elastic-agents goes offline and get back online status frequently Elastic Agent	6	886	December 19, 2023
Fleet Server Agent Communication issue Elastic Agent fleet	1	401	July 19, 2023

Fail to checkin to fleet server

Related topics