Fleet server goes offline right after successfull install

Hi,
i followed the steps in kibana and the docs to install a fleet server on premise (generated certificates from the elasticsearch CA),

curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.12.0-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.12.0-linux-x86_64.tar.gz
cd elastic-agent-8.12.0-linux-x86_64
sudo ./elastic-agent install --url=https://10.11.200.40:8220 \
   --fleet-server-es=https://192.168.37.209:9200 \
   --fleet-server-service-token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX \
   --fleet-server-policy=fleet-server-policy \
   --certificate-authorities=/home/antd/certs/fleet-ca/ca.crt \
   --fleet-server-es-ca=/home/antd/certs/http_ca.crt \
   --fleet-server-cert=/home/antd/certs/fleet-server/fleet-server.crt \
   --fleet-server-cert-key=/home/antd/certs/fleet-server/fleet-server.key \
   --fleet-server-port=8220

The install seems to go right and the agent is installed,

Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:
[    ] Service Started  [14s] Elastic Agent successfully installed, starting enrollment.
[=   ] Waiting For Enroll...  [16s] {"log.level":"info","@timestamp":"2024-03-21T19:14:53.434Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
[ ===] Waiting For Enroll...  [18s] {"log.level":"info","@timestamp":"2024-03-21T19:14:55.437Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":787},"message":"Waiting for Elastic Agent to start","ecs.version":"1.6.0"}
[ ===] Waiting For Enroll...  [22s] {"log.level":"info","@timestamp":"2024-03-21T19:14:59.443Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":818},"message":"Fleet Server - Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}
[ ===] Waiting For Enroll...  [23s] {"log.level":"info","@timestamp":"2024-03-21T19:15:00.258Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Starting enrollment to URL: https://10.11.200.40:8220/","ecs.version":"1.6.0"}
[=   ] Waiting For Enroll...  [24s] {"log.level":"info","@timestamp":"2024-03-21T19:15:01.418Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-21T19:15:01.419Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":285},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[=   ] Done  [24s]                               
Elastic Agent has been successfully installed.

but on kibana the agent shows as offline and updating

and when looking at the logs i see some errors relating to version compatibility but i have 8.12.0 on my kibana and elasticsearch nodes.

root@elastic04:~# /opt/Elastic/Agent/elastic-agent logs
{"log.level":"info","@timestamp":"2024-03-21T19:38:36.135Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed fleet-server-default (FAILED->STARTING): Starting","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"STARTING","old_state":"FAILED"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-21T19:38:36.135Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed fleet-server-default-fleet-server-fleet_server-c45bee43-160f-49cb-8e48-008eb847bc14 (FAILED->STARTING): Starting","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server-fleet_server-c45bee43-160f-49cb-8e48-008eb847bc14","type":"input","state":"STARTING","old_state":"FAILED"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-21T19:38:36.141Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","error.message":"Forbidden","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-21T19:38:36.141Z","message":"Stats endpoint (/opt/Elastic/Agent/data/tmp/UwGGXFL1il700DVAc6q-T-1Z9J1UjGMU.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/UwGGXFL1il700DVAc6q-T-1Z9J1UjGMU.sock: use of closed network connection","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","log.logger":"fleet-metrics.api","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-21T19:38:36.141Z","message":"Error - failed version compatibility check with elasticsearch: Forbidden","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"FAILED","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-21T19:38:36.141Z","message":"Fleet Server failed","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"failed version compatibility check with elasticsearch: Forbidden","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-21T19:38:36.142Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed fleet-server-default (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: Forbidden","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-21T19:38:36.142Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed fleet-server-default-fleet-server-fleet_server-c45bee43-160f-49cb-8e48-008eb847bc14 (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: Forbidden","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server-fleet_server-c45bee43-160f-49cb-8e48-008eb847bc14","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-21T19:38:36.257Z","message":"Non-zero metrics in the last 30s","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"log.logger":"monitoring","log.origin":{"file.line":187,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"service.name":"filebeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":1880691006}},"memory":{"mem":{"usage":{"bytes":809934848}}}},"cpu":{"system":{"ticks":760,"time":{"ms":10}},"total":{"ticks":2890,"time":{"ms":30},"value":2890},"user":{"ticks":2130,"time":{"ms":20}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":22},"info":{"ephemeral_id":"3e14e972-af34-400b-b94f-db0c7df73c23","uptime":{"ms":1410119},"version":"8.12.0"},"memstats":{"gc_next":71522688,"memory_alloc":38464792,"memory_total":175544840,"rss":136028160},"runtime":{"goroutines":75}},"filebeat":{"events":{"active":3206},"harvester":{"open_files":4,"running":4}},"libbeat":{"config":{"module":{"running":2}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":3202}}},"registrar":{"states":{"current":2}},"system":{"load":{"1":0.02,"15":0.1,"5":0.08,"norm":{"1":0.0004,"15":0.0018,"5":0.0014}}}}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-21T19:38:36.544Z","message":"Non-zero metrics in the last 30s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":1880543599}},"memory":{"mem":{"usage":{"bytes":809832448}}}},"cpu":{"system":{"ticks":10740,"time":{"ms":20}},"total":{"ticks":27400,"time":{"ms":170},"value":27400},"user":{"ticks":16660,"time":{"ms":150}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":15},"info":{"ephemeral_id":"062af8f9-aa80-4f17-8148-88466f9bf0ed","uptime":{"ms":1410133},"version":"8.12.0"},"memstats":{"gc_next":95652432,"memory_alloc":47501936,"memory_total":1689296432,"rss":159772672},"runtime":{"goroutines":76}},"libbeat":{"config":{"module":{"running":11}},"output":{"events":{"active":0},"read":{"bytes":1995},"write":{"bytes":99}},"pipeline":{"clients":11,"events":{"active":3211,"retry":42}}},"system":{"load":{"1":0.02,"15":0.1,"5":0.08,"norm":{"1":0.0004,"15":0.0018,"5":0.0014}}}}},"log.logger":"monitoring","log.origin":{"file.line":187,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"service.name":"metricbeat","ecs.version":"1.6.0"}

root@elasticsearch-master:~# /usr/share/elasticsearch/bin/elasticsearch --version
Version: 8.12.0, Build: deb/1665f706fd9354802c02146c1e6b5c0fbcddfbc9/2024-01-11T10:05:27.953830042Z, JVM: 21.0.1

mheni@kibana:~$ /usr/share/kibana/bin/kibana --version
{"log.level":"info","@timestamp":"2024-03-21T20:11:24.939Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","agentVersion":"4.2.0","env":{"pid":20893,"proctitle":"/usr/share/kibana/bin/../node/bin/node","os":"linux 4.15.0-213-generic","arch":"x64","host":"kibana","timezone":"UTC+00","runtime":"Node.js v18.18.2"},"config":{"active":{"source":"start","value":true},"breakdownMetrics":{"source":"start","value":false},"captureBody":{"source":"start","value":"off","commonName":"capture_body"},"captureHeaders":{"source":"start","value":false},"centralConfig":{"source":"start","value":false},"contextPropagationOnly":{"source":"start","value":true},"environment":{"source":"start","value":"production"},"globalLabels":{"source":"start","value":[["git_rev","e9092c0a17923f4ed984456b8a5db619b0a794b3"]],"sourceValue":{"git_rev":"e9092c0a17923f4ed984456b8a5db619b0a794b3"}},"logLevel":{"source":"default","value":"info","commonName":"log_level"},"metricsInterval":{"source":"start","value":120,"sourceValue":"120s"},"serverUrl":{"source":"start","value":"https://kibana-cloud-apm.apm.us-east-1.aws.found.io/","commonName":"server_url"},"transactionSampleRate":{"source":"start","value":0.1,"commonName":"transaction_sample_rate"},"captureSpanStackTraces":{"source":"start","sourceValue":false},"secretToken":{"source":"start","value":"[REDACTED]","commonName":"secret_token"},"serviceName":{"source":"start","value":"kibana","commonName":"service_name"},"serviceVersion":{"source":"start","value":"8.12.0","commonName":"service_version"}},"activationMethod":"require","message":"Elastic APM Node.js Agent v4.2.0"}
8.12.0

and the journalctl logs show some errors as well looks like it's trying to get openstack and azure metadata for some reason?

...
Mar 21 19:14:52 elastic04 elastic-agent[38248]: {"log.level":"warn","@timestamp":"2024-03-21T19:14:52.986Z","message":"Performance preset 'balanced' overrides user setting for field 'bulk_max_size'","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"elasticsearch","log.origin":{"file.line":66,"file.name":"elasticsearch/elasticsearch.go","function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.makeES"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
...
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"warn","@timestamp":"2024-03-21T19:14:53.025Z","message":"Performance preset 'balanced' overrides user setting for field 'bulk_max_size'","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"elasticsearch","log.origin":{"file.line":66,"file.name":"elasticsearch/elasticsearch.go","function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.makeES"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
...
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"warn","@timestamp":"2024-03-21T19:14:53.135Z","message":"Performance preset 'balanced' overrides user setting for field 'bulk_max_size'","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"elasticsearch","log.origin":{"file.line":66,"file.name":"elasticsearch/elasticsearch.go","function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.makeES"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.136Z","message":"elasticsearch url: http://127.0.0.1:9200","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.origin":{"file.line":122,"file.name":"eslegclient/connection.go","function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnection"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"esclientleg","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.138Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed beat/metrics-monitoring (STARTING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"beat/metrics-monitoring","type":"output","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.140Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed beat/metrics-monitoring-metrics-monitoring-beats (STARTING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"beat/metrics-monitoring-metrics-monitoring-beats","type":"input","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.435Z","log.origin":{"file.name":"cmd/run.go","file.line":303},"message":"reexec shutdown channel triggered","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.435Z","message":"Received signal \"terminated\", stopping","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"service","log.origin":{"file.line":52,"file.name":"service/service.go","function":"github.com/elastic/elastic-agent-libs/service.HandleSignals.func1"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.436Z","message":"Received signal \"terminated\", stopping","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"service","log.origin":{"file.line":52,"file.name":"service/service.go","function":"github.com/elastic/elastic-agent-libs/service.HandleSignals.func1"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.435Z","message":"Received signal \"terminated\", stopping","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"service","log.origin":{"file.line":52,"file.name":"service/service.go","function":"github.com/elastic/elastic-agent-libs/service.HandleSignals.func1"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.436Z","message":"Stopping 1 runners ...","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.origin":{"file.line":188,"file.name":"cfgfile/list.go","function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Stop"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"centralmgmt","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.436Z","message":"Received signal \"terminated\", stopping","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"service","log.origin":{"file.line":52,"file.name":"service/service.go","function":"github.com/elastic/elastic-agent-libs/service.HandleSignals.func1"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.436Z","message":"Stopping 4 runners ...","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"centralmgmt","log.origin":{"file.line":188,"file.name":"cfgfile/list.go","function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Stop"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.436Z","message":"Stopping filebeat","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":484,"file.name":"beater/filebeat.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Stop"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.436Z","message":"Stopping 1 runners ...","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"centralmgmt","log.origin":{"file.line":188,"file.name":"cfgfile/list.go","function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Stop"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Total metrics","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"monitoring","log.origin":{"file.line":195,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logTotals"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":0}},"id":"elastic-agent.service","stats":{"periods":0,"throttled":{"ns":0,"periods":0}}},"cpuacct":{"id":"elastic-agent.service","total":{"ns":2389889087}},"memory":{"id":"elastic-agent.service","mem":{"limit":{"bytes":9223372036854772000},"usage":{"bytes":233926656}}}},"cpu":{"system":{"ticks":70,"time":{"ms":70}},"total":{"ticks":270,"time":{"ms":270},"value":270},"user":{"ticks":200,"time":{"ms":200}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":20},"info":{"ephemeral_id":"a5bc451e-cfc6-4e0e-9d44-bf95bf2299ed","name":"metricbeat","uptime":{"ms":991},"version":"8.12.0"},"memstats":{"gc_next":47012192,"memory_alloc":35782320,"memory_sys":61207816,"memory_total":53215904,"rss":123940864},"runtime":{"goroutines":48}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops":0},"reloads":0,"scans":0},"output":{"batches":{"split":0},"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":0,"retry":0,"total":0},"queue":{"acked":0,"max_events":0}}},"processor":{"add_host_metadata":{"fqdn_lookup_failed":0}},"system":{"cpu":{"cores":56},"load":{"1":0.42,"15":0.18,"5":0.29,"norm":{"1":0.0075,"15":0.0032,"5":0.0052}}}}},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Uptime: 1.000010285s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"monitoring","log.origin":{"file.line":196,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logTotals"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Stopping metrics logging.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"monitoring","log.origin":{"file.line":163,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Stats endpoint (/opt/Elastic/Agent/data/tmp/akSPbdqgaHaTY0_J01-dsfYK6JpMz2zn.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/akSPbdqgaHaTY0_J01-dsfYK6JpMz2zn.sock: use of closed network connection","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0","log.logger":"api","log.origin":{"file.line":73,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1"},"service.name":"metricbeat","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"metricbeat stopped.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0","log.origin":{"file.line":532,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"service.name":"metricbeat","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Total metrics","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"monitoring","log.origin":{"file.line":195,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logTotals"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":0}},"id":"elastic-agent.service","stats":{"periods":0,"throttled":{"ns":0,"periods":0}}},"cpuacct":{"id":"elastic-agent.service","total":{"ns":2389241599}},"memory":{"id":"elastic-agent.service","mem":{"limit":{"bytes":9223372036854772000},"usage":{"bytes":233926656}}}},"cpu":{"system":{"ticks":90,"time":{"ms":90}},"total":{"ticks":290,"time":{"ms":290},"value":290},"user":{"ticks":200,"time":{"ms":200}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":22},"info":{"ephemeral_id":"54723a0a-4716-4106-98f2-ae737399622f","name":"metricbeat","uptime":{"ms":1468},"version":"8.12.0"},"memstats":{"gc_next":46992680,"memory_alloc":37271648,"memory_sys":69334280,"memory_total":53839168,"rss":128090112},"runtime":{"goroutines":55}},"libbeat":{"config":{"module":{"running":1,"starts":1,"stops":0},"reloads":0,"scans":0},"output":{"batches":{"split":0},"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":0,"retry":0,"total":0},"queue":{"acked":0,"max_events":3200}}},"processor":{"add_host_metadata":{"fqdn_lookup_failed":0}},"system":{"cpu":{"cores":56},"load":{"1":0.42,"15":0.18,"5":0.29,"norm":{"1":0.0075,"15":0.0032,"5":0.0052}}}}},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Uptime: 1.474665853s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"monitoring","log.origin":{"file.line":196,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logTotals"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Stopping metrics logging.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.origin":{"file.line":163,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"monitoring","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Stats endpoint (/opt/Elastic/Agent/data/tmp/Hk6rvk9TDibMPcDvpl0jkLE-qDsHWVYL.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/Hk6rvk9TDibMPcDvpl0jkLE-qDsHWVYL.sock: use of closed network connection","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"api","log.origin":{"file.line":73,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"metricbeat stopped.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.origin":{"file.line":532,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"error","@timestamp":"2024-03-21T19:14:54.920Z","message":"add_cloud_metadata: received error failed requesting azure metadata: Get \"http://169.254.169.254/metadata/instance/compute?api-version=2021-02-01\": dial tcp 169.254.169.254:80: i/o timeout","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.920Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":100,"file.name":"add_cloud_metadata/add_cloud_metadata.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.925Z","message":"Total metrics","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"monitoring","log.origin":{"file.line":195,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logTotals"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":0}},"id":"elastic-agent.service","stats":{"periods":0,"throttled":{"ns":0,"periods":0}}},"cpuacct":{"id":"elastic-agent.service","total":{"ns":2448806599}},"memory":{"id":"elastic-agent.service","mem":{"limit":{"bytes":9223372036854772000},"usage":{"bytes":133124096}}}},"cpu":{"system":{"ticks":150,"time":{"ms":150}},"total":{"ticks":320,"time":{"ms":320},"value":320},"user":{"ticks":170,"time":{"ms":170}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":13},"info":{"ephemeral_id":"da312df3-1df4-43e0-a49e-49a70c65a65f","name":"metricbeat","uptime":{"ms":3097},"version":"8.12.0"},"memstats":{"gc_next":47130912,"memory_alloc":38494992,"memory_sys":65139976,"memory_total":55022352,"rss":130158592},"runtime":{"goroutines":27}},"libbeat":{"config":{"module":{"running":4,"starts":4,"stops":0},"reloads":0,"scans":0},"output":{"batches":{"split":0},"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":6,"filtered":0,"published":0,"retry":0,"total":6},"queue":{"acked":0,"max_events":3200}}},"processor":{"add_host_metadata":{"fqdn_lookup_failed":0}},"system":{"cpu":{"cores":56},"load":{"1":0.42,"15":0.18,"5":0.29,"norm":{"1":0.0075,"15":0.0032,"5":0.0052}}}}},"ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.925Z","message":"Uptime: 3.102610009s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"monitoring","log.origin":{"file.line":196,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logTotals"},"ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.925Z","message":"Stopping metrics logging.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":163,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"monitoring","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.925Z","message":"Stats endpoint (/opt/Elastic/Agent/data/tmp/iThI_df0cBKC6YUNGGlKscMkOfz3FBH3.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/iThI_df0cBKC6YUNGGlKscMkOfz3FBH3.sock: use of closed network connection","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"api","log.origin":{"file.line":73,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.925Z","message":"metricbeat stopped.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":532,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"error","@timestamp":"2024-03-21T19:14:54.994Z","message":"add_cloud_metadata: received error failed requesting openstack metadata: Get \"https://169.254.169.254/2009-04-04/meta-data/instance-id\": dial tcp 169.254.169.254:443: i/o timeout","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"filebeat","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"error","@timestamp":"2024-03-21T19:14:54.994Z","message":"add_cloud_metadata: received error failed requesting openstack metadata: Get \"http://169.254.169.254/2009-04-04/meta-data/instance-id\": dial tcp 169.254.169.254:80: i/o timeout","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":100,"file.name":"add_cloud_metadata/add_cloud_metadata.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1"},"service.name":"filebeat","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"unregistering","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"metric_registry","log.origin":{"file.line":70,"file.name":"inputmon/input.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/inputmon.NewInputRegistry.func1"},"input_type":"filestream","id":"filestream-monitoring-agent","key":"filestream-monitoring-agent","uuid":"35c7a712-1f72-4185-a1d2-8c70f13d4154","ecs.version":"1.6.0"}
Mar 21 19:14:55 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"Input 'filestream' stopped (goroutine)","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"input.filestream","log.origin":{"file.line":134,"file.name":"compat/compat.go","function":"github.com/elastic/beats/v7/filebeat/input/v2/compat.(*runner).Start.func1"},"service.name":"filebeat","id":"filestream-monitoring-agent","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:55 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"Input 'filestream' stopped (runner)","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","id":"filestream-monitoring-agent","ecs.version":"1.6.0","log.logger":"input.filestream","log.origin":{"file.line":142,"file.name":"compat/compat.go","function":"github.com/elastic/beats/v7/filebeat/input/v2/compat.(*runner).Stop"},"ecs.version":"1.6.0"}
Mar 21 19:14:55 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"Stopping Crawler","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":155,"file.name":"beater/crawler.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Stop"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:55 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"Stopping 0 inputs","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":165,"file.name":"beater/crawler.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Stop"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:55 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"Crawler stopped","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":185,"file.name":"beater/crawler.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Stop"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:55 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"Stopping Registrar","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":130,"file.name":"registrar/registrar.go","function":"github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Stop"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"registrar","ecs.version":"1.6.0"}
...

Any ideas ?

Not sure if this is relevant but we do have an HTTP proxy that egress traffic to the internet has to pass through, i mention this because i see some "Forbiden" messages in the logs. but the traffic between the fleet server and kibana and elasticsearch is not required to go through the proxy.

And this is the status i get from the elastic-agent CLI
Not sure why it's trying to use port 8221 when the install command specifies 8220

root@elastic04:~# /opt/Elastic/Agent/elastic-agent status
┌─ fleet
│  └─ status: (FAILED) fail to checkin to fleet-server: all hosts failed: 1 error occurred:
│         * requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/8404d019-3e40-4ae0-9385-f3f7207cfd05/checkin?": dial tcp 127.0.0.1:8221: connect: connection refused
│     
│     
└─ elastic-agent
   ├─ status: (DEGRADED) 1 or more components/units in a failed state
   └─ fleet-server-default
      ├─ status: (HEALTHY) Healthy: communicating with pid '38491'
      ├─ fleet-server-default
      │  └─ status: (FAILED) Error - failed version compatibility check with elasticsearch: Forbidden
      └─ fleet-server-default-fleet-server-fleet_server-c45bee43-160f-49cb-8e48-008eb847bc14
         └─ status: (FAILED) Error - failed version compatibility check with elasticsearch: Forbidden

Also i can confirm that i can reach the elasticsearch cluster from this server

root@elastic04:~# curl -k -u elastic_user:password --cacert /home/antd/certs/http_ca.crt https://192.168.37.209:9200 
{
  "name" : "es-master",
  "cluster_name" : "nstap",
  "cluster_uuid" : "_2uf4C6FRcS50d3ELnLZVw",
  "version" : {
    "number" : "8.12.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "1665f706fd9354802c02146c1e6b5c0fbcddfbc9",
    "build_date" : "2024-01-11T10:05:27.953830042Z",
    "build_snapshot" : false,
    "lucene_version" : "9.9.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Hello,

Yeah it looks like the forbidden error prevents fleet-server to run properly. Can the fleet-server host access elasticsearch?

Hi Julia,
Thanks for the relpy.

Yes the fleet host can access Elasticsearch, i tested using curl with the username, password, and HTTP certificate.

you can see the response i got in my previous reply.

it also works without the -k curl option

root@elastic04:~# curl -u elastic_user:password --cacert /home/antd/certs/http_ca.crt https://192.168.37.14:9200/_cat/nodes
192.168.38.28  54 58 0 0.00 0.00 0.00 w  - es-warm-02
192.168.37.209 10 74 2 0.00 0.44 0.51 dm * es-master
192.168.38.234 47 59 0 0.00 0.00 0.00 w  - es-warm-01
192.168.37.14  22 65 0 0.00 0.01 0.00 hm - es-hot-01
192.168.37.249  1 62 0 0.11 0.18 0.12 h  - es-hot-02

Your fleet-server url seems to be https://10.11.200.40:8220 , is this correct? Is it the same as elastic04 host?

yes,
elastic04 is the host on which I want to install fleet, and that's its IP.

I'm a bit confused about the elastic-agent install command and whether it installs fleet server and then installs the agent or is it expecting that fleet is already installed on that URL.

i assumed that it is installing it since that is the command i got from kibana when clicking on add fleet server button

Yes, the command from Add Fleet Server button should install agent with a fleet-server.

Hi Julia, Thanks for the clarification.

i was able to get rid of the forbidden errors, by removing the proxy config from fleet server settings on kibana.

{"log.level":"error","@timestamp":"2024-03-28T13:55:12.667Z","message":"Failed to connect to backoff(elasticsearch(https://192.168.37.209:9200)): Get \"https://192.168.37.209:9200\": Forbidden","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"udp-default","type":"udp"},"log":{"source":"udp-default"},"ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"filebeat","ecs.version":"1.6.0"}

and now the elastic-agent and fleet server are showing a healthy status when using the CLI.

root@elastic04:~# /opt/Elastic/Agent/elastic-agent status
┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   └─ status: (HEALTHY) Running

root@elastic04:~# ss -tunlp | grep -i 'elastic'
tcp     LISTEN   0        4096            127.0.0.1:6789          0.0.0.0:*      users:(("elastic-agent",pid=118007,fd=14))                                     
tcp     LISTEN   0        4096            127.0.0.1:6791          0.0.0.0:*      users:(("elastic-agent",pid=118007,fd=15))                                     
root@elastic04:~# ss -tunlp | grep -i 'fleet'
tcp     LISTEN   0        4096            127.0.0.1:8221          0.0.0.0:*      users:(("fleet-server",pid=119111,fd=15))                                      
tcp     LISTEN   0        4096                    *:8220                *:*      users:(("fleet-server",pid=119111,fd=9))  

however on kibana the status still shows offline

looking at the logs i see an issue with the certificate, can it be configured to bypass SSL verfication or to trust self signed certs?

root@elastic04:~# /opt/Elastic/Agent/elastic-agent logs
{"log.level":"error","@timestamp":"2024-03-28T13:57:32.409Z","message":"Failed to connect to backoff(elasticsearch(https://192.168.37.209:9200)): Get \"https://192.168.37.209:9200\": x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T13:57:32.409Z","message":"Attempting to reconnect to backoff(elasticsearch(https://192.168.37.209:9200)) with 4 reconnect attempt(s)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":139,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-28T13:57:32.431Z","message":"Error dialing x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"esclientleg","log.origin":{"file.line":38,"file.name":"transport/logging.go","function":"github.com/elastic/elastic-agent-libs/transport.LoggingDialer.func1"},"ecs.version":"1.6.0","service.name":"metricbeat","network":"tcp","address":"192.168.37.209:9200","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T13:57:33.137Z","message":"Running on policy with Fleet Server integration: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-28T13:57:35.010Z","message":"Failed to connect to backoff(elasticsearch(https://192.168.37.209:9200)): Get \"https://192.168.37.209:9200\": x509: certificate signed by unknown authority","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T13:57:35.010Z","message":"Attempting to reconnect to backoff(elasticsearch(https://192.168.37.209:9200)) with 4 reconnect attempt(s)","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":139,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-28T13:57:35.036Z","message":"Error dialing x509: certificate signed by unknown authority","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"esclientleg","ecs.version":"1.6.0","service.name":"filebeat","network":"tcp","address":"192.168.37.209:9200","log.origin":{"file.line":38,"file.name":"transport/logging.go","function":"github.com/elastic/elastic-agent-libs/transport.LoggingDialer.func1"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-28T13:57:35.894Z","message":"Failed to connect to backoff(elasticsearch(https://192.168.37.209:9200)): Get \"https://192.168.37.209:9200\": x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T13:57:35.894Z","message":"Attempting to reconnect to backoff(elasticsearch(https://192.168.37.209:9200)) with 4 reconnect attempt(s)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.origin":{"file.line":139,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","ecs.version":"1.6.0"}

Yes, you can use the insecure flag to bypass SSL verification: Troubleshoot common problems | Fleet and Elastic Agent Guide [8.13] | Elastic

Thanks for the link,

the --insecure option helped eliminate the certificate errors, but the issue persists with healthy (connected) status when using the cli, but kibana still says it's offline

root@elastic04:~# /opt/Elastic/Agent/elastic-agent status
┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   └─ status: (HEALTHY) Running

the logs look good overall, the only errors i still get are when it tries to get metadata for openstack, hetzner, huawei ...etc. from this static IP 169.254.169.254.

is this necessary for an on premise deployment? can it be skipped?

{"log.level":"info","@timestamp":"2024-03-28T15:50:34.840Z","message":"Applying performance preset 'balanced': {\n  \"bulk_max_size\": 1600,\n  \"compression_level\": 1,\n  \"idle_connection_timeout\": \"3s\",\n  \"queue\": {\n    \"mem\": {\n      \"events\": 3200,\n      \"flush\": {\n        \"min_events\": 1600,\n        \"timeout\": \"10s\"\n      }\n    }\n  },\n  \"worker\": 1\n}","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"elasticsearch","log.origin":{"file.line":63,"file.name":"elasticsearch/elasticsearch.go","function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.makeES"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-03-28T15:50:34.840Z","message":"Performance preset 'balanced' overrides user setting for field 'bulk_max_size'","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"elasticsearch","log.origin":{"file.line":66,"file.name":"elasticsearch/elasticsearch.go","function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.makeES"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:34.840Z","message":"elasticsearch url: https://192.168.37.209:9200","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"esclientleg","log.origin":{"file.line":122,"file.name":"eslegclient/connection.go","function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnection"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:34.841Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed beat/metrics-monitoring (STARTING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"beat/metrics-monitoring","type":"output","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:34.852Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed beat/metrics-monitoring-metrics-monitoring-beats (STARTING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"beat/metrics-monitoring-metrics-monitoring-beats","type":"input","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-28T15:50:35.611Z","message":"add_cloud_metadata: received error failed requesting openstack metadata: Get \"http://169.254.169.254/2009-04-04/meta-data/hostname\": dial tcp 169.254.169.254:80: i/o timeout","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:35.611Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":100,"file.name":"add_cloud_metadata/add_cloud_metadata.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:35.831Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"udp-default","type":"udp"},"log":{"source":"udp-default"},"ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":100,"file.name":"add_cloud_metadata/add_cloud_metadata.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-28T15:50:36.211Z","message":"add_cloud_metadata: received error failed requesting openstack metadata: Get \"https://169.254.169.254/2009-04-04/meta-data/hostname\": dial tcp 169.254.169.254:443: i/o timeout","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"tcp-default","type":"tcp"},"log":{"source":"tcp-default"},"ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-28T15:50:36.211Z","message":"add_cloud_metadata: received error failed requesting hetzner metadata: Get \"http://169.254.169.254/hetzner/v1/metadata/instance-id\": dial tcp 169.254.169.254:80: i/o timeout","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"tcp-default","type":"tcp"},"log":{"source":"tcp-default"},"log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-28T15:50:36.211Z","message":"add_cloud_metadata: received error failed requesting huawei metadata: Get \"http://169.254.169.254/openstack/latest/meta_data.json\": dial tcp 169.254.169.254:80: i/o timeout","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"tcp-default","type":"tcp"},"log":{"source":"tcp-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-28T15:50:36.211Z","message":"add_cloud_metadata: received error failed requesting openstack metadata: Get \"http://169.254.169.254/2009-04-04/meta-data/placement/availability-zone\": dial tcp 169.254.169.254:80: i/o timeout","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"tcp-default","type":"tcp"},"log":{"source":"tcp-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:36.211Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"tcp-default","type":"tcp"},"log":{"source":"tcp-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":100,"file.name":"add_cloud_metadata/add_cloud_metadata.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-28T15:50:36.295Z","message":"add_cloud_metadata: received error failed requesting gcp metadata: Get \"http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json\": dial tcp 169.254.169.254:80: i/o timeout","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:36.296Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":100,"file.name":"add_cloud_metadata/add_cloud_metadata.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-28T15:50:36.398Z","message":"add_cloud_metadata: received error failed requesting gcp metadata: Get \"http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json\": dial tcp 169.254.169.254:80: i/o timeout","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:36.398Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":100,"file.name":"add_cloud_metadata/add_cloud_metadata.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-28T15:50:36.511Z","message":"add_cloud_metadata: received error failed requesting digitalocean metadata: Get \"http://169.254.169.254/metadata/v1.json\": dial tcp 169.254.169.254:80: i/o timeout","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:36.511Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":100,"file.name":"add_cloud_metadata/add_cloud_metadata.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-28T15:50:36.782Z","message":"add_cloud_metadata: received error failed requesting huawei metadata: Get \"http://169.254.169.254/openstack/latest/meta_data.json\": dial tcp 169.254.169.254:80: i/o timeout","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:36.782Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":100,"file.name":"add_cloud_metadata/add_cloud_metadata.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:38.213Z","message":"Running on policy with Fleet Server integration: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:38.214Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed fleet-server-default-fleet-server-fleet_server-c45bee43-160f-49cb-8e48-008eb847bc14 (CONFIGURING->HEALTHY): Running on policy with Fleet Server integration: fleet-server-policy","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server-fleet_server-c45bee43-160f-49cb-8e48-008eb847bc14","type":"input","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:38.214Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed fleet-server-default (CONFIGURING->HEALTHY): Running on policy with Fleet Server integration: fleet-server-policy","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:43.473Z","message":"Running on policy with Fleet Server integration: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:45.612Z","message":"Connecting to backoff(elasticsearch(https://192.168.37.209:9200))","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":137,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:45.674Z","message":"Attempting to connect to Elasticsearch version 8.12.0 (default)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"esclientleg","log.origin":{"file.line":304,"file.name":"eslegclient/connection.go","function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:45.680Z","message":"Connection to backoff(elasticsearch(https://192.168.37.209:9200)) established","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":145,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:45.832Z","message":"Connecting to backoff(elasticsearch(https://192.168.37.209:9200))","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"udp-default","type":"udp"},"log":{"source":"udp-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":137,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:45.885Z","message":"Attempting to connect to Elasticsearch version 8.12.0 (default)","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"udp-default","type":"udp"},"log":{"source":"udp-default"},"log.logger":"esclientleg","log.origin":{"file.line":304,"file.name":"eslegclient/connection.go","function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:45.889Z","message":"Connection to backoff(elasticsearch(https://192.168.37.209:9200)) established","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"udp-default","type":"udp"},"log":{"source":"udp-default"},"log.origin":{"file.line":145,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:46.400Z","message":"Connecting to backoff(elasticsearch(https://192.168.37.209:9200))","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":137,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:46.463Z","message":"Attempting to connect to Elasticsearch version 8.12.0 (default)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"esclientleg","log.origin":{"file.line":304,"file.name":"eslegclient/connection.go","function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:46.468Z","message":"Connection to backoff(elasticsearch(https://192.168.37.209:9200)) established","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.origin":{"file.line":145,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:46.514Z","message":"Connecting to backoff(elasticsearch(https://192.168.37.209:9200))","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":137,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:46.564Z","message":"Attempting to connect to Elasticsearch version 8.12.0 (default)","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"esclientleg","log.origin":{"file.line":304,"file.name":"eslegclient/connection.go","function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:46.567Z","message":"Connection to backoff(elasticsearch(https://192.168.37.209:9200)) established","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":145,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:46.785Z","message":"Connecting to backoff(elasticsearch(https://192.168.37.209:9200))","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":137,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:46.837Z","message":"Attempting to connect to Elasticsearch version 8.12.0 (default)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"esclientleg","log.origin":{"file.line":304,"file.name":"eslegclient/connection.go","function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:46.843Z","message":"Connection to backoff(elasticsearch(https://192.168.37.209:9200)) established","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":145,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:48.732Z","message":"Running on policy with Fleet Server integration: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:53.769Z","message":"Running on policy with Fleet Server integration: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"state":"HEALTHY","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:50:59.029Z","message":"Running on policy with Fleet Server integration: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:51:02.656Z","message":"Non-zero metrics in the last 30s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"elastic-agent.service"},"cpuacct":{"id":"elastic-agent.service","total":{"ns":47214671196}},"memory":{"id":"elastic-agent.service","mem":{"limit":{"bytes":9223372036854772000},"usage":{"bytes":459108352}}}},"cpu":{"system":{"ticks":390,"time":{"ms":390}},"total":{"ticks":1070,"time":{"ms":1070},"value":1070},"user":{"ticks":680,"time":{"ms":680}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":15},"info":{"ephemeral_id":"a1e8fd79-d93f-4543-9901-d5541d76d1f2","name":"metricbeat","uptime":{"ms":30130},"version":"8.12.0"},"memstats":{"gc_next":55365840,"memory_alloc":29991912,"memory_sys":78181640,"memory_total":110178360,"rss":144015360},"runtime":{"goroutines":76}},"libbeat":{"config":{"module":{"running":11,"starts":11}},"output":{"events":{"acked":74,"active":0,"batches":2,"total":74},"read":{"bytes":4387,"errors":2},"type":"elasticsearch","write":{"bytes":25858}},"pipeline":{"clients":11,"events":{"active":25,"filtered":3,"published":99,"retry":43,"total":102},"queue":{"acked":74,"max_events":3200}}},"metricbeat":{"system":{"cpu":{"events":3,"success":3},"diskio":{"events":29,"success":29},"filesystem":{"events":6,"success":6},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":26,"success":26},"process":{"events":22,"success":22},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"cpu":{"cores":56},"load":{"1":0.29,"15":0.14,"5":0.14,"norm":{"1":0.0052,"15":0.0025,"5":0.0025}}}}},"log.logger":"monitoring","log.origin":{"file.line":187,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:51:02.852Z","message":"Non-zero metrics in the last 30s","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"udp-default","type":"udp"},"log":{"source":"udp-default"},"log.logger":"monitoring","log.origin":{"file.line":187,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"service.name":"filebeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"elastic-agent.service"},"cpuacct":{"id":"elastic-agent.service","total":{"ns":47221528305}},"memory":{"id":"elastic-agent.service","mem":{"limit":{"bytes":9223372036854772000},"usage":{"bytes":459239424}}}},"cpu":{"system":{"ticks":190,"time":{"ms":190}},"total":{"ticks":650,"time":{"ms":650},"value":650},"user":{"ticks":460,"time":{"ms":460}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":19},"info":{"ephemeral_id":"3042c0b0-5778-47aa-be18-6ce72836aa5d","name":"filebeat","uptime":{"ms":30117},"version":"8.12.0"},"memstats":{"gc_next":42643488,"memory_alloc":31282072,"memory_sys":61207816,"memory_total":87161008,"rss":114376704},"runtime":{"goroutines":46}},"filebeat":{"events":{"active":166,"added":685,"done":519},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1,"starts":1}},"output":{"events":{"acked":519,"active":0,"batches":2,"total":519},"read":{"bytes":15490,"errors":2},"type":"elasticsearch","write":{"bytes":47026}},"pipeline":{"clients":1,"events":{"active":165,"published":684,"retry":278,"total":684},"queue":{"acked":519,"max_events":3200}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":56},"load":{"1":0.29,"15":0.14,"5":0.14,"norm":{"1":0.0052,"15":0.0025,"5":0.0025}}}}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:51:03.225Z","message":"Non-zero metrics in the last 30s","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"tcp-default","type":"tcp"},"log":{"source":"tcp-default"},"log.origin":{"file.line":187,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"service.name":"filebeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"elastic-agent.service"},"cpuacct":{"id":"elastic-agent.service","total":{"ns":47237359404}},"memory":{"id":"elastic-agent.service","mem":{"limit":{"bytes":9223372036854772000},"usage":{"bytes":459370496}}}},"cpu":{"system":{"ticks":60,"time":{"ms":60}},"total":{"ticks":300,"time":{"ms":300},"value":300},"user":{"ticks":240,"time":{"ms":240}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":19},"info":{"ephemeral_id":"54e175c0-6e57-43a4-acc2-d064bc5b3506","name":"filebeat","uptime":{"ms":30118},"version":"8.12.0"},"memstats":{"gc_next":37021976,"memory_alloc":25385920,"memory_sys":61732104,"memory_total":67344408,"rss":111378432},"runtime":{"goroutines":46}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":1,"starts":1}},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0},"queue":{"max_events":3200}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":56},"load":{"1":0.29,"15":0.14,"5":0.14,"norm":{"1":0.0052,"15":0.0025,"5":0.0025}}}}},"log.logger":"monitoring","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:51:03.312Z","message":"Non-zero metrics in the last 30s","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"service.name":"filebeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"elastic-agent.service"},"cpuacct":{"id":"elastic-agent.service","total":{"ns":47243809421}},"memory":{"id":"elastic-agent.service","mem":{"limit":{"bytes":9223372036854772000},"usage":{"bytes":459370496}}}},"cpu":{"system":{"ticks":90,"time":{"ms":90}},"total":{"ticks":280,"time":{"ms":280},"value":280},"user":{"ticks":190,"time":{"ms":190}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":18},"info":{"ephemeral_id":"7fb2c738-5e4e-4ac8-acaf-0e918b2891ea","name":"filebeat","uptime":{"ms":30113},"version":"8.12.0"},"memstats":{"gc_next":37480944,"memory_alloc":25536240,"memory_sys":61994248,"memory_total":67866808,"rss":114991104},"runtime":{"goroutines":52}},"filebeat":{"events":{"active":0,"added":4,"done":4},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":2,"starts":2}},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":2,"events":{"active":0,"filtered":4,"total":4},"queue":{"max_events":3200}}},"registrar":{"states":{"current":4,"update":4},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":56},"load":{"1":0.29,"15":0.14,"5":0.14,"norm":{"1":0.0052,"15":0.0025,"5":0.0025}}}}},"log.logger":"monitoring","log.origin":{"file.line":187,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:51:03.441Z","message":"Non-zero metrics in the last 30s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"monitoring","log.origin":{"file.line":187,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"elastic-agent.service"},"cpuacct":{"id":"elastic-agent.service","total":{"ns":47254854609}},"memory":{"id":"elastic-agent.service","mem":{"limit":{"bytes":9223372036854772000},"usage":{"bytes":459485184}}}},"cpu":{"system":{"ticks":120,"time":{"ms":120}},"total":{"ticks":410,"time":{"ms":410},"value":410},"user":{"ticks":290,"time":{"ms":290}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":21},"info":{"ephemeral_id":"753e3994-f346-4bfb-a640-90ae3f11369a","name":"metricbeat","uptime":{"ms":30130},"version":"8.12.0"},"memstats":{"gc_next":51776032,"memory_alloc":27980416,"memory_sys":73856264,"memory_total":63061440,"rss":135290880},"runtime":{"goroutines":83}},"libbeat":{"config":{"module":{"running":9,"starts":9}},"output":{"events":{"acked":8,"active":0,"batches":1,"total":8},"read":{"bytes":2538,"errors":1},"type":"elasticsearch","write":{"bytes":4738}},"pipeline":{"clients":9,"events":{"active":0,"published":8,"retry":8,"total":8},"queue":{"acked":8,"max_events":3200}}},"metricbeat":{"http":{"json":{"events":8,"success":8}}},"system":{"cpu":{"cores":56},"load":{"1":0.29,"15":0.14,"5":0.14,"norm":{"1":0.0052,"15":0.0025,"5":0.0025}}}}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:51:03.522Z","message":"Non-zero metrics in the last 30s","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"monitoring","log.origin":{"file.line":187,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"service.name":"filebeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"elastic-agent.service"},"cpuacct":{"id":"elastic-agent.service","total":{"ns":47263301061}},"memory":{"id":"elastic-agent.service","mem":{"limit":{"bytes":9223372036854772000},"usage":{"bytes":459599872}}}},"cpu":{"system":{"ticks":80,"time":{"ms":80}},"total":{"ticks":430,"time":{"ms":430},"value":430},"user":{"ticks":350,"time":{"ms":350}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":20},"info":{"ephemeral_id":"a88161c2-c955-4f19-abeb-2c084e746111","name":"filebeat","uptime":{"ms":30102},"version":"8.12.0"},"memstats":{"gc_next":44269992,"memory_alloc":30229536,"memory_sys":64877832,"memory_total":83883240,"rss":116940800},"runtime":{"goroutines":60}},"filebeat":{"events":{"active":2,"added":468,"done":466},"harvester":{"open_files":6,"running":6,"started":6}},"libbeat":{"config":{"module":{"running":1,"starts":1}},"output":{"events":{"acked":323,"active":0,"batches":2,"total":323},"read":{"bytes":10393,"errors":2},"type":"elasticsearch","write":{"bytes":71985}},"pipeline":{"clients":6,"events":{"active":2,"filtered":143,"published":325,"retry":316,"total":468},"queue":{"acked":323,"max_events":3200}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":56},"load":{"1":0.29,"15":0.14,"5":0.14,"norm":{"1":0.0052,"15":0.0025,"5":0.0025}}}}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:51:03.821Z","message":"Non-zero metrics in the last 30s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"monitoring","log.origin":{"file.line":187,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"elastic-agent.service"},"cpuacct":{"id":"elastic-agent.service","total":{"ns":47335683764}},"memory":{"id":"elastic-agent.service","mem":{"limit":{"bytes":9223372036854772000},"usage":{"bytes":459771904}}}},"cpu":{"system":{"ticks":90,"time":{"ms":90}},"total":{"ticks":350,"time":{"ms":350},"value":350},"user":{"ticks":260,"time":{"ms":260}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":22},"info":{"ephemeral_id":"e8732d21-1dac-4abe-8013-cf9b88cb7632","name":"metricbeat","uptime":{"ms":30137},"version":"8.12.0"},"memstats":{"gc_next":46393680,"memory_alloc":41082800,"memory_sys":70448392,"memory_total":60827856,"rss":129904640},"runtime":{"goroutines":89}},"libbeat":{"config":{"module":{"running":5,"starts":5}},"output":{"events":{"acked":2,"active":0,"batches":1,"total":2},"read":{"bytes":2381,"errors":1},"type":"elasticsearch","write":{"bytes":3252}},"pipeline":{"clients":10,"events":{"active":0,"published":2,"retry":2,"total":2},"queue":{"acked":2,"max_events":3200}}},"metricbeat":{"beat":{"state":{"events":1,"success":1},"stats":{"events":1,"success":1}}},"system":{"cpu":{"cores":56},"load":{"1":0.29,"15":0.14,"5":0.14,"norm":{"1":0.0052,"15":0.0025,"5":0.0025}}}}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:51:04.289Z","message":"Running on policy with Fleet Server integration: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:51:09.549Z","message":"Running on policy with Fleet Server integration: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"state":"HEALTHY","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-28T15:51:14.563Z","message":"Running on policy with Fleet Server integration: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}

This processor is added by default to all Beats when running under Elastic-Agent and it cannot be disabled.
If you don't need the data from this processor, the error can be ignored, it should not affect the Beat health.

The agent being offline is strange, is there any check-in related error in agent logs? Based on the agent status it doesn't look unhealthy. When you re-enroll an agent, it gets a new id, so the old agent will show up as offline. Do you not see any new agent as online?

Hi Julia,

yes i don't see any new agents online, just one offline agent.

but when looking at the Data streams, it looks like some data is coming in from the installed integrations? i don't see any indices created for juniper-srx for example but maybe i need to look into the integration configuration.

does this look like what i'm supposed to see when everything is installed correctly? maybe there is just an issue with kibana and it's showing it as offline?

It's possible that the agent doesn't check-in that frequently or has some failed checkins. The UI shows it checked in 20 minutes ago (offline is shown when the last checkin was more than 5m ago, it doesn't actually mean that the agent is offline for good). The default check-in interval is 30s, perhaps it was changed?

I would say if the data is coming in as expected, then the agent looks working.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.