Hi,
i followed the steps in kibana and the docs to install a fleet server on premise (generated certificates from the elasticsearch CA),
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.12.0-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.12.0-linux-x86_64.tar.gz
cd elastic-agent-8.12.0-linux-x86_64
sudo ./elastic-agent install --url=https://10.11.200.40:8220 \
--fleet-server-es=https://192.168.37.209:9200 \
--fleet-server-service-token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX \
--fleet-server-policy=fleet-server-policy \
--certificate-authorities=/home/antd/certs/fleet-ca/ca.crt \
--fleet-server-es-ca=/home/antd/certs/http_ca.crt \
--fleet-server-cert=/home/antd/certs/fleet-server/fleet-server.crt \
--fleet-server-cert-key=/home/antd/certs/fleet-server/fleet-server.key \
--fleet-server-port=8220
The install seems to go right and the agent is installed,
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:
[ ] Service Started [14s] Elastic Agent successfully installed, starting enrollment.
[= ] Waiting For Enroll... [16s] {"log.level":"info","@timestamp":"2024-03-21T19:14:53.434Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
[ ===] Waiting For Enroll... [18s] {"log.level":"info","@timestamp":"2024-03-21T19:14:55.437Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":787},"message":"Waiting for Elastic Agent to start","ecs.version":"1.6.0"}
[ ===] Waiting For Enroll... [22s] {"log.level":"info","@timestamp":"2024-03-21T19:14:59.443Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":818},"message":"Fleet Server - Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}
[ ===] Waiting For Enroll... [23s] {"log.level":"info","@timestamp":"2024-03-21T19:15:00.258Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":496},"message":"Starting enrollment to URL: https://10.11.200.40:8220/","ecs.version":"1.6.0"}
[= ] Waiting For Enroll... [24s] {"log.level":"info","@timestamp":"2024-03-21T19:15:01.418Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":461},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-21T19:15:01.419Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":285},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[= ] Done [24s]
Elastic Agent has been successfully installed.
but on kibana the agent shows as offline and updating
and when looking at the logs i see some errors relating to version compatibility but i have 8.12.0 on my kibana and elasticsearch nodes.
root@elastic04:~# /opt/Elastic/Agent/elastic-agent logs
{"log.level":"info","@timestamp":"2024-03-21T19:38:36.135Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed fleet-server-default (FAILED->STARTING): Starting","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"STARTING","old_state":"FAILED"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-21T19:38:36.135Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed fleet-server-default-fleet-server-fleet_server-c45bee43-160f-49cb-8e48-008eb847bc14 (FAILED->STARTING): Starting","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server-fleet_server-c45bee43-160f-49cb-8e48-008eb847bc14","type":"input","state":"STARTING","old_state":"FAILED"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-21T19:38:36.141Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","error.message":"Forbidden","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-21T19:38:36.141Z","message":"Stats endpoint (/opt/Elastic/Agent/data/tmp/UwGGXFL1il700DVAc6q-T-1Z9J1UjGMU.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/UwGGXFL1il700DVAc6q-T-1Z9J1UjGMU.sock: use of closed network connection","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","log.logger":"fleet-metrics.api","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-21T19:38:36.141Z","message":"Error - failed version compatibility check with elasticsearch: Forbidden","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"FAILED","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-21T19:38:36.141Z","message":"Fleet Server failed","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"failed version compatibility check with elasticsearch: Forbidden","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-21T19:38:36.142Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed fleet-server-default (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: Forbidden","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-03-21T19:38:36.142Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed fleet-server-default-fleet-server-fleet_server-c45bee43-160f-49cb-8e48-008eb847bc14 (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: Forbidden","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server-fleet_server-c45bee43-160f-49cb-8e48-008eb847bc14","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-21T19:38:36.257Z","message":"Non-zero metrics in the last 30s","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"log.logger":"monitoring","log.origin":{"file.line":187,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"service.name":"filebeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":1880691006}},"memory":{"mem":{"usage":{"bytes":809934848}}}},"cpu":{"system":{"ticks":760,"time":{"ms":10}},"total":{"ticks":2890,"time":{"ms":30},"value":2890},"user":{"ticks":2130,"time":{"ms":20}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":22},"info":{"ephemeral_id":"3e14e972-af34-400b-b94f-db0c7df73c23","uptime":{"ms":1410119},"version":"8.12.0"},"memstats":{"gc_next":71522688,"memory_alloc":38464792,"memory_total":175544840,"rss":136028160},"runtime":{"goroutines":75}},"filebeat":{"events":{"active":3206},"harvester":{"open_files":4,"running":4}},"libbeat":{"config":{"module":{"running":2}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":3202}}},"registrar":{"states":{"current":2}},"system":{"load":{"1":0.02,"15":0.1,"5":0.08,"norm":{"1":0.0004,"15":0.0018,"5":0.0014}}}}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-21T19:38:36.544Z","message":"Non-zero metrics in the last 30s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":1880543599}},"memory":{"mem":{"usage":{"bytes":809832448}}}},"cpu":{"system":{"ticks":10740,"time":{"ms":20}},"total":{"ticks":27400,"time":{"ms":170},"value":27400},"user":{"ticks":16660,"time":{"ms":150}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":15},"info":{"ephemeral_id":"062af8f9-aa80-4f17-8148-88466f9bf0ed","uptime":{"ms":1410133},"version":"8.12.0"},"memstats":{"gc_next":95652432,"memory_alloc":47501936,"memory_total":1689296432,"rss":159772672},"runtime":{"goroutines":76}},"libbeat":{"config":{"module":{"running":11}},"output":{"events":{"active":0},"read":{"bytes":1995},"write":{"bytes":99}},"pipeline":{"clients":11,"events":{"active":3211,"retry":42}}},"system":{"load":{"1":0.02,"15":0.1,"5":0.08,"norm":{"1":0.0004,"15":0.0018,"5":0.0014}}}}},"log.logger":"monitoring","log.origin":{"file.line":187,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"},"service.name":"metricbeat","ecs.version":"1.6.0"}
root@elasticsearch-master:~# /usr/share/elasticsearch/bin/elasticsearch --version
Version: 8.12.0, Build: deb/1665f706fd9354802c02146c1e6b5c0fbcddfbc9/2024-01-11T10:05:27.953830042Z, JVM: 21.0.1
mheni@kibana:~$ /usr/share/kibana/bin/kibana --version
{"log.level":"info","@timestamp":"2024-03-21T20:11:24.939Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","agentVersion":"4.2.0","env":{"pid":20893,"proctitle":"/usr/share/kibana/bin/../node/bin/node","os":"linux 4.15.0-213-generic","arch":"x64","host":"kibana","timezone":"UTC+00","runtime":"Node.js v18.18.2"},"config":{"active":{"source":"start","value":true},"breakdownMetrics":{"source":"start","value":false},"captureBody":{"source":"start","value":"off","commonName":"capture_body"},"captureHeaders":{"source":"start","value":false},"centralConfig":{"source":"start","value":false},"contextPropagationOnly":{"source":"start","value":true},"environment":{"source":"start","value":"production"},"globalLabels":{"source":"start","value":[["git_rev","e9092c0a17923f4ed984456b8a5db619b0a794b3"]],"sourceValue":{"git_rev":"e9092c0a17923f4ed984456b8a5db619b0a794b3"}},"logLevel":{"source":"default","value":"info","commonName":"log_level"},"metricsInterval":{"source":"start","value":120,"sourceValue":"120s"},"serverUrl":{"source":"start","value":"https://kibana-cloud-apm.apm.us-east-1.aws.found.io/","commonName":"server_url"},"transactionSampleRate":{"source":"start","value":0.1,"commonName":"transaction_sample_rate"},"captureSpanStackTraces":{"source":"start","sourceValue":false},"secretToken":{"source":"start","value":"[REDACTED]","commonName":"secret_token"},"serviceName":{"source":"start","value":"kibana","commonName":"service_name"},"serviceVersion":{"source":"start","value":"8.12.0","commonName":"service_version"}},"activationMethod":"require","message":"Elastic APM Node.js Agent v4.2.0"}
8.12.0
and the journalctl logs show some errors as well looks like it's trying to get openstack and azure metadata for some reason?
...
Mar 21 19:14:52 elastic04 elastic-agent[38248]: {"log.level":"warn","@timestamp":"2024-03-21T19:14:52.986Z","message":"Performance preset 'balanced' overrides user setting for field 'bulk_max_size'","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"elasticsearch","log.origin":{"file.line":66,"file.name":"elasticsearch/elasticsearch.go","function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.makeES"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
...
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"warn","@timestamp":"2024-03-21T19:14:53.025Z","message":"Performance preset 'balanced' overrides user setting for field 'bulk_max_size'","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"elasticsearch","log.origin":{"file.line":66,"file.name":"elasticsearch/elasticsearch.go","function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.makeES"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
...
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"warn","@timestamp":"2024-03-21T19:14:53.135Z","message":"Performance preset 'balanced' overrides user setting for field 'bulk_max_size'","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"elasticsearch","log.origin":{"file.line":66,"file.name":"elasticsearch/elasticsearch.go","function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.makeES"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.136Z","message":"elasticsearch url: http://127.0.0.1:9200","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.origin":{"file.line":122,"file.name":"eslegclient/connection.go","function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnection"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"esclientleg","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.138Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed beat/metrics-monitoring (STARTING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"beat/metrics-monitoring","type":"output","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.140Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":601},"message":"Unit state changed beat/metrics-monitoring-metrics-monitoring-beats (STARTING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"beat/metrics-monitoring-metrics-monitoring-beats","type":"input","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.435Z","log.origin":{"file.name":"cmd/run.go","file.line":303},"message":"reexec shutdown channel triggered","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.435Z","message":"Received signal \"terminated\", stopping","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"service","log.origin":{"file.line":52,"file.name":"service/service.go","function":"github.com/elastic/elastic-agent-libs/service.HandleSignals.func1"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.436Z","message":"Received signal \"terminated\", stopping","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"service","log.origin":{"file.line":52,"file.name":"service/service.go","function":"github.com/elastic/elastic-agent-libs/service.HandleSignals.func1"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.435Z","message":"Received signal \"terminated\", stopping","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"service","log.origin":{"file.line":52,"file.name":"service/service.go","function":"github.com/elastic/elastic-agent-libs/service.HandleSignals.func1"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.436Z","message":"Stopping 1 runners ...","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.origin":{"file.line":188,"file.name":"cfgfile/list.go","function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Stop"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"centralmgmt","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.436Z","message":"Received signal \"terminated\", stopping","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"service","log.origin":{"file.line":52,"file.name":"service/service.go","function":"github.com/elastic/elastic-agent-libs/service.HandleSignals.func1"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.436Z","message":"Stopping 4 runners ...","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"centralmgmt","log.origin":{"file.line":188,"file.name":"cfgfile/list.go","function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Stop"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.436Z","message":"Stopping filebeat","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":484,"file.name":"beater/filebeat.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Stop"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.436Z","message":"Stopping 1 runners ...","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"centralmgmt","log.origin":{"file.line":188,"file.name":"cfgfile/list.go","function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Stop"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Total metrics","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"monitoring","log.origin":{"file.line":195,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logTotals"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":0}},"id":"elastic-agent.service","stats":{"periods":0,"throttled":{"ns":0,"periods":0}}},"cpuacct":{"id":"elastic-agent.service","total":{"ns":2389889087}},"memory":{"id":"elastic-agent.service","mem":{"limit":{"bytes":9223372036854772000},"usage":{"bytes":233926656}}}},"cpu":{"system":{"ticks":70,"time":{"ms":70}},"total":{"ticks":270,"time":{"ms":270},"value":270},"user":{"ticks":200,"time":{"ms":200}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":20},"info":{"ephemeral_id":"a5bc451e-cfc6-4e0e-9d44-bf95bf2299ed","name":"metricbeat","uptime":{"ms":991},"version":"8.12.0"},"memstats":{"gc_next":47012192,"memory_alloc":35782320,"memory_sys":61207816,"memory_total":53215904,"rss":123940864},"runtime":{"goroutines":48}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops":0},"reloads":0,"scans":0},"output":{"batches":{"split":0},"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":0,"retry":0,"total":0},"queue":{"acked":0,"max_events":0}}},"processor":{"add_host_metadata":{"fqdn_lookup_failed":0}},"system":{"cpu":{"cores":56},"load":{"1":0.42,"15":0.18,"5":0.29,"norm":{"1":0.0075,"15":0.0032,"5":0.0052}}}}},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Uptime: 1.000010285s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"monitoring","log.origin":{"file.line":196,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logTotals"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Stopping metrics logging.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"monitoring","log.origin":{"file.line":163,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Stats endpoint (/opt/Elastic/Agent/data/tmp/akSPbdqgaHaTY0_J01-dsfYK6JpMz2zn.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/akSPbdqgaHaTY0_J01-dsfYK6JpMz2zn.sock: use of closed network connection","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0","log.logger":"api","log.origin":{"file.line":73,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1"},"service.name":"metricbeat","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"metricbeat stopped.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0","log.origin":{"file.line":532,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"service.name":"metricbeat","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Total metrics","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"monitoring","log.origin":{"file.line":195,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logTotals"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":0}},"id":"elastic-agent.service","stats":{"periods":0,"throttled":{"ns":0,"periods":0}}},"cpuacct":{"id":"elastic-agent.service","total":{"ns":2389241599}},"memory":{"id":"elastic-agent.service","mem":{"limit":{"bytes":9223372036854772000},"usage":{"bytes":233926656}}}},"cpu":{"system":{"ticks":90,"time":{"ms":90}},"total":{"ticks":290,"time":{"ms":290},"value":290},"user":{"ticks":200,"time":{"ms":200}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":22},"info":{"ephemeral_id":"54723a0a-4716-4106-98f2-ae737399622f","name":"metricbeat","uptime":{"ms":1468},"version":"8.12.0"},"memstats":{"gc_next":46992680,"memory_alloc":37271648,"memory_sys":69334280,"memory_total":53839168,"rss":128090112},"runtime":{"goroutines":55}},"libbeat":{"config":{"module":{"running":1,"starts":1,"stops":0},"reloads":0,"scans":0},"output":{"batches":{"split":0},"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":0,"retry":0,"total":0},"queue":{"acked":0,"max_events":3200}}},"processor":{"add_host_metadata":{"fqdn_lookup_failed":0}},"system":{"cpu":{"cores":56},"load":{"1":0.42,"15":0.18,"5":0.29,"norm":{"1":0.0075,"15":0.0032,"5":0.0052}}}}},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Uptime: 1.474665853s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"monitoring","log.origin":{"file.line":196,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logTotals"},"ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Stopping metrics logging.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.origin":{"file.line":163,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"monitoring","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"Stats endpoint (/opt/Elastic/Agent/data/tmp/Hk6rvk9TDibMPcDvpl0jkLE-qDsHWVYL.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/Hk6rvk9TDibMPcDvpl0jkLE-qDsHWVYL.sock: use of closed network connection","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"api","log.origin":{"file.line":73,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:53 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:53.441Z","message":"metricbeat stopped.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.origin":{"file.line":532,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"error","@timestamp":"2024-03-21T19:14:54.920Z","message":"add_cloud_metadata: received error failed requesting azure metadata: Get \"http://169.254.169.254/metadata/instance/compute?api-version=2021-02-01\": dial tcp 169.254.169.254:80: i/o timeout","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.920Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":100,"file.name":"add_cloud_metadata/add_cloud_metadata.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.925Z","message":"Total metrics","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"monitoring","log.origin":{"file.line":195,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logTotals"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":0}},"id":"elastic-agent.service","stats":{"periods":0,"throttled":{"ns":0,"periods":0}}},"cpuacct":{"id":"elastic-agent.service","total":{"ns":2448806599}},"memory":{"id":"elastic-agent.service","mem":{"limit":{"bytes":9223372036854772000},"usage":{"bytes":133124096}}}},"cpu":{"system":{"ticks":150,"time":{"ms":150}},"total":{"ticks":320,"time":{"ms":320},"value":320},"user":{"ticks":170,"time":{"ms":170}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":13},"info":{"ephemeral_id":"da312df3-1df4-43e0-a49e-49a70c65a65f","name":"metricbeat","uptime":{"ms":3097},"version":"8.12.0"},"memstats":{"gc_next":47130912,"memory_alloc":38494992,"memory_sys":65139976,"memory_total":55022352,"rss":130158592},"runtime":{"goroutines":27}},"libbeat":{"config":{"module":{"running":4,"starts":4,"stops":0},"reloads":0,"scans":0},"output":{"batches":{"split":0},"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":6,"filtered":0,"published":0,"retry":0,"total":6},"queue":{"acked":0,"max_events":3200}}},"processor":{"add_host_metadata":{"fqdn_lookup_failed":0}},"system":{"cpu":{"cores":56},"load":{"1":0.42,"15":0.18,"5":0.29,"norm":{"1":0.0075,"15":0.0032,"5":0.0052}}}}},"ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.925Z","message":"Uptime: 3.102610009s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"monitoring","log.origin":{"file.line":196,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logTotals"},"ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.925Z","message":"Stopping metrics logging.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":163,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"monitoring","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.925Z","message":"Stats endpoint (/opt/Elastic/Agent/data/tmp/iThI_df0cBKC6YUNGGlKscMkOfz3FBH3.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/iThI_df0cBKC6YUNGGlKscMkOfz3FBH3.sock: use of closed network connection","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"api","log.origin":{"file.line":73,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.925Z","message":"metricbeat stopped.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":532,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"error","@timestamp":"2024-03-21T19:14:54.994Z","message":"add_cloud_metadata: received error failed requesting openstack metadata: Get \"https://169.254.169.254/2009-04-04/meta-data/instance-id\": dial tcp 169.254.169.254:443: i/o timeout","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"filebeat","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"error","@timestamp":"2024-03-21T19:14:54.994Z","message":"add_cloud_metadata: received error failed requesting openstack metadata: Get \"http://169.254.169.254/2009-04-04/meta-data/instance-id\": dial tcp 169.254.169.254:80: i/o timeout","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":173,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"add_cloud_metadata","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"add_cloud_metadata: hosting provider type not detected.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"ecs.version":"1.6.0","log.logger":"add_cloud_metadata","log.origin":{"file.line":100,"file.name":"add_cloud_metadata/add_cloud_metadata.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1"},"service.name":"filebeat","ecs.version":"1.6.0"}
Mar 21 19:14:54 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"unregistering","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"metric_registry","log.origin":{"file.line":70,"file.name":"inputmon/input.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/inputmon.NewInputRegistry.func1"},"input_type":"filestream","id":"filestream-monitoring-agent","key":"filestream-monitoring-agent","uuid":"35c7a712-1f72-4185-a1d2-8c70f13d4154","ecs.version":"1.6.0"}
Mar 21 19:14:55 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"Input 'filestream' stopped (goroutine)","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"input.filestream","log.origin":{"file.line":134,"file.name":"compat/compat.go","function":"github.com/elastic/beats/v7/filebeat/input/v2/compat.(*runner).Start.func1"},"service.name":"filebeat","id":"filestream-monitoring-agent","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:55 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"Input 'filestream' stopped (runner)","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","id":"filestream-monitoring-agent","ecs.version":"1.6.0","log.logger":"input.filestream","log.origin":{"file.line":142,"file.name":"compat/compat.go","function":"github.com/elastic/beats/v7/filebeat/input/v2/compat.(*runner).Stop"},"ecs.version":"1.6.0"}
Mar 21 19:14:55 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"Stopping Crawler","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":155,"file.name":"beater/crawler.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Stop"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:55 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"Stopping 0 inputs","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":165,"file.name":"beater/crawler.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Stop"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:55 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"Crawler stopped","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":185,"file.name":"beater/crawler.go","function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Stop"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Mar 21 19:14:55 elastic04 elastic-agent[38248]: {"log.level":"info","@timestamp":"2024-03-21T19:14:54.994Z","message":"Stopping Registrar","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.origin":{"file.line":130,"file.name":"registrar/registrar.go","function":"github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Stop"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"registrar","ecs.version":"1.6.0"}
...
Any ideas ?
Not sure if this is relevant but we do have an HTTP proxy that egress traffic to the internet has to pass through, i mention this because i see some "Forbiden" messages in the logs. but the traffic between the fleet server and kibana and elasticsearch is not required to go through the proxy.
