I cant seem to receive any logs from elastic agent installed on either Windows or Ubuntu machine. I've searched through the documentations to the best of my abilities but I'm afraid that I'm still too new to Elastic.
My current setup is a simple lab - just a single VM that host Elasticsearch, kibana, logstash, filebeat, and also fleet server. The Elastic agent is installed in a different Windows and Ubuntu VM.
Here's what I've tried/found out so far:
Doing tcpdump on Elastic server, I can see that the Windows and Linux machines are sending something to 8220 and 9200. The server always replies with a RST packet when receiving on 9200.
Both of the Elastic Agents can register perfectly and the status is shown as "Healthy".
I'm using self-signed certificate and already using "--insecure" option when registering.
I can receive logs from agent installed in fleet server.
Here's the logs (I think I got the correct log file?)
root@eleong-test-endpoint:/opt/Elastic/Agent# tail -80 elastic-agent-20211124221553
2021-11-24T22:15:57.325+0800 DEBUG status/reporter.go:200 'operator-default-bafebdad' has status 'online'
2021-11-24T22:15:57.325+0800 DEBUG status/reporter.go:212 'filebeat--7.15.1-36361854' has status 'online'
2021-11-24T22:15:57.325+0800 DEBUG status/reporter.go:212 'metricbeat--7.15.1-3d432d01' has status 'online'
2021-11-24T22:15:57.325+0800 DEBUG status/reporter.go:212 'filebeat--7.15.1--36643631373035623733363936343635-c2b62aed' has status 'online'
2021-11-24T22:15:57.325+0800 DEBUG status/reporter.go:212 'metricbeat--7.15.1--36643631373035623733363936343635-d81e9a56' has status 'online'
2021-11-24T22:15:57.325+0800 DEBUG operation/operator.go:293 operator is looking for filebeat--7.15.1 in app collection: map[filebeat--7.15.1:0xc0000d4500 filebeat--7.15.1--36643631373035623733363936343635:0xc0000d4640 metricbeat--7.15.1:0xc0000d4140 metricbeat--7.15.1--36643631373035623733363936343635:0xc0001d8dc0]
2021-11-24T22:15:57.325+0800 DEBUG operation/operation_fetch.go:61 binary 'filebeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/filebeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:57.325+0800 DEBUG operation/operator.go:273 running operation 'retryable block: operation-fetch operation-verify' for filebeat.7.15.1
2021-11-24T22:15:57.325+0800 DEBUG operation/operation_fetch.go:61 binary 'filebeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/filebeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:57.326+0800 DEBUG operation/operation_retryable.go:83 running operation 'operation-verify' of the block 'retryable block: operation-fetch operation-verify'
2021-11-24T22:15:57.542+0800 DEBUG status/reporter.go:200 'capabilities-e3c763c7' has status 'online'
2021-11-24T22:15:57.543+0800 DEBUG status/reporter.go:200 'operator-default-bafebdad' has status 'online'
2021-11-24T22:15:57.543+0800 DEBUG status/reporter.go:212 'filebeat--7.15.1-36361854' has status 'online'
2021-11-24T22:15:57.543+0800 DEBUG status/reporter.go:212 'metricbeat--7.15.1-3d432d01' has status 'online'
2021-11-24T22:15:57.543+0800 DEBUG status/reporter.go:212 'filebeat--7.15.1--36643631373035623733363936343635-c2b62aed' has status 'online'
2021-11-24T22:15:57.543+0800 DEBUG status/reporter.go:212 'metricbeat--7.15.1--36643631373035623733363936343635-d81e9a56' has status 'online'
2021-11-24T22:15:57.543+0800 INFO log/reporter.go:40 2021-11-24T22:15:57+08:00 - message: Application: filebeat--7.15.1--36643631373035623733363936343635[282120f3-ddba-450f-8661-a04c4ea6c731]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
2021-11-24T22:15:57.609+0800 INFO [api] api/server.go:62 Starting stats endpoint
2021-11-24T22:15:57.610+0800 INFO application/managed_mode.go:291 Agent is starting
2021-11-24T22:15:57.610+0800 INFO [api] api/server.go:64 Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
2021-11-24T22:15:57.904+0800 INFO operation/operator.go:269 operation 'operation-install' skipped for filebeat.7.15.1
2021-11-24T22:15:57.904+0800 INFO operation/operator.go:269 operation 'operation-start' skipped for filebeat.7.15.1
2021-11-24T22:15:57.904+0800 DEBUG operation/operator.go:273 running operation 'operation-config' for filebeat.7.15.1
2021-11-24T22:15:57.907+0800 DEBUG operation/operator.go:293 operator is looking for metricbeat--7.15.1 in app collection: map[filebeat--7.15.1:0xc0000d4500 filebeat--7.15.1--36643631373035623733363936343635:0xc0000d4640 metricbeat--7.15.1:0xc0000d4140 metricbeat--7.15.1--36643631373035623733363936343635:0xc0001d8dc0]
2021-11-24T22:15:57.907+0800 DEBUG operation/operation_fetch.go:61 binary 'metricbeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/metricbeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:57.907+0800 DEBUG operation/operator.go:273 running operation 'retryable block: operation-fetch operation-verify' for metricbeat.7.15.1
2021-11-24T22:15:57.907+0800 DEBUG operation/operation_fetch.go:61 binary 'metricbeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/metricbeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:57.907+0800 DEBUG operation/operation_retryable.go:83 running operation 'operation-verify' of the block 'retryable block: operation-fetch operation-verify'
2021-11-24T22:15:58.032+0800 DEBUG fleet/fleet_gateway.go:160 FleetGateway calling Checkin API
2021-11-24T22:15:58.032+0800 DEBUG fleet/fleet_gateway.go:202 Checking started
2021-11-24T22:15:58.134+0800 DEBUG fleet/fleet_gateway.go:233 using previously saved ack token: 842b5f13-1993-429a-ab75-e02c9ad7c97d
2021-11-24T22:15:58.134+0800 DEBUG remote/client.go:169 Request method: POST, path: /api/fleet/agents/282120f3-ddba-450f-8661-a04c4ea6c731/checkin
2021-11-24T22:15:58.357+0800 INFO operation/operator.go:269 operation 'operation-install' skipped for metricbeat.7.15.1
2021-11-24T22:15:58.357+0800 INFO operation/operator.go:269 operation 'operation-start' skipped for metricbeat.7.15.1
2021-11-24T22:15:58.357+0800 DEBUG operation/operator.go:273 running operation 'operation-config' for metricbeat.7.15.1
2021-11-24T22:15:58.387+0800 DEBUG operation/operator.go:293 operator is looking for filebeat--7.15.1--36643631373035623733363936343635 in app collection: map[filebeat--7.15.1:0xc0000d4500 filebeat--7.15.1--36643631373035623733363936343635:0xc0000d4640 metricbeat--7.15.1:0xc0000d4140 metricbeat--7.15.1--36643631373035623733363936343635:0xc0001d8dc0]
2021-11-24T22:15:58.387+0800 DEBUG operation/operation_fetch.go:61 binary 'filebeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/filebeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:58.387+0800 DEBUG operation/operator.go:273 running operation 'retryable block: operation-fetch operation-verify' for filebeat.7.15.1
2021-11-24T22:15:58.387+0800 DEBUG operation/operation_fetch.go:61 binary 'filebeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/filebeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:58.389+0800 DEBUG operation/operation_retryable.go:83 running operation 'operation-verify' of the block 'retryable block: operation-fetch operation-verify'
2021-11-24T22:15:58.645+0800 INFO operation/operator.go:269 operation 'operation-install' skipped for filebeat.7.15.1
2021-11-24T22:15:58.645+0800 INFO operation/operator.go:269 operation 'operation-start' skipped for filebeat.7.15.1
2021-11-24T22:15:58.645+0800 DEBUG operation/operator.go:273 running operation 'operation-config' for filebeat.7.15.1
2021-11-24T22:15:58.647+0800 DEBUG operation/operator.go:293 operator is looking for metricbeat--7.15.1--36643631373035623733363936343635 in app collection: map[filebeat--7.15.1:0xc0000d4500 filebeat--7.15.1--36643631373035623733363936343635:0xc0000d4640 metricbeat--7.15.1:0xc0000d4140 metricbeat--7.15.1--36643631373035623733363936343635:0xc0001d8dc0]
2021-11-24T22:15:58.647+0800 DEBUG operation/operation_fetch.go:61 binary 'metricbeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/metricbeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:58.647+0800 DEBUG operation/operator.go:273 running operation 'retryable block: operation-fetch operation-verify' for metricbeat.7.15.1
2021-11-24T22:15:58.647+0800 DEBUG operation/operation_fetch.go:61 binary 'metricbeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/metricbeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:58.647+0800 DEBUG operation/operation_retryable.go:83 running operation 'operation-verify' of the block 'retryable block: operation-fetch operation-verify'
2021-11-24T22:15:58.781+0800 DEBUG status/reporter.go:200 'operator-default-bafebdad' has status 'online'
2021-11-24T22:15:58.781+0800 DEBUG status/reporter.go:200 'gateway-0521b85b' has status 'online'
2021-11-24T22:15:58.781+0800 DEBUG status/reporter.go:200 'capabilities-e3c763c7' has status 'online'
2021-11-24T22:15:58.781+0800 DEBUG status/reporter.go:212 'filebeat--7.15.1-36361854' has status 'online'
2021-11-24T22:15:58.781+0800 DEBUG status/reporter.go:212 'metricbeat--7.15.1-3d432d01' has status 'online'
2021-11-24T22:15:58.781+0800 DEBUG status/reporter.go:212 'filebeat--7.15.1--36643631373035623733363936343635-c2b62aed' has status 'online'
2021-11-24T22:15:58.781+0800 DEBUG status/reporter.go:212 'metricbeat--7.15.1--36643631373035623733363936343635-d81e9a56' has status 'online'
2021-11-24T22:15:58.781+0800 INFO log/reporter.go:40 2021-11-24T22:15:58+08:00 - message: Application: metricbeat--7.15.1--36643631373035623733363936343635[282120f3-ddba-450f-8661-a04c4ea6c731]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
2021-11-24T22:15:58.853+0800 INFO operation/operator.go:269 operation 'operation-install' skipped for metricbeat.7.15.1
2021-11-24T22:15:58.853+0800 INFO operation/operator.go:269 operation 'operation-start' skipped for metricbeat.7.15.1
2021-11-24T22:15:58.853+0800 DEBUG operation/operator.go:273 running operation 'operation-config' for metricbeat.7.15.1
2021-11-24T22:15:58.856+0800 DEBUG status/reporter.go:200 'gateway-0521b85b' has status 'online'
2021-11-24T22:15:58.856+0800 DEBUG status/reporter.go:200 'capabilities-e3c763c7' has status 'online'
2021-11-24T22:15:58.856+0800 DEBUG status/reporter.go:200 'operator-default-bafebdad' has status 'online'
2021-11-24T22:15:58.856+0800 DEBUG status/reporter.go:212 'filebeat--7.15.1--36643631373035623733363936343635-c2b62aed' has status 'online'
2021-11-24T22:15:58.856+0800 DEBUG status/reporter.go:212 'metricbeat--7.15.1--36643631373035623733363936343635-d81e9a56' has status 'online'
2021-11-24T22:15:58.856+0800 DEBUG status/reporter.go:212 'filebeat--7.15.1-36361854' has status 'online'
2021-11-24T22:15:58.856+0800 DEBUG status/reporter.go:212 'metricbeat--7.15.1-3d432d01' has status 'online'
2021-11-24T22:15:58.856+0800 INFO stateresolver/stateresolver.go:66 Updating internal state
2021-11-24T22:20:48.622+0800 DEBUG dispatcher/dispatcher.go:79 No action to dispatch
2021-11-24T22:20:48.623+0800 DEBUG fleet/fleet_gateway.go:184 FleetGateway is sleeping, next update in 1s
2021-11-24T22:20:48.623+0800 DEBUG status/reporter.go:200 'capabilities-e3c763c7' has status 'online'
2021-11-24T22:20:48.623+0800 DEBUG status/reporter.go:200 'operator-default-bafebdad' has status 'online'
2021-11-24T22:20:48.623+0800 DEBUG status/reporter.go:200 'gateway-0521b85b' has status 'online'
2021-11-24T22:20:48.623+0800 DEBUG status/reporter.go:212 'filebeat--7.15.1-36361854' has status 'online'
2021-11-24T22:20:48.623+0800 DEBUG status/reporter.go:212 'metricbeat--7.15.1-3d432d01' has status 'online'
2021-11-24T22:20:48.623+0800 DEBUG status/reporter.go:212 'filebeat--7.15.1--36643631373035623733363936343635-c2b62aed' has status 'online'
2021-11-24T22:20:48.623+0800 DEBUG status/reporter.go:212 'metricbeat--7.15.1--36643631373035623733363936343635-d81e9a56' has status 'online'
2021-11-24T22:20:49.671+0800 DEBUG fleet/fleet_gateway.go:160 FleetGateway calling Checkin API
2021-11-24T22:20:49.671+0800 DEBUG fleet/fleet_gateway.go:202 Checking started
2021-11-24T22:20:49.773+0800 DEBUG fleet/fleet_gateway.go:233 using previously saved ack token: 842b5f13-1993-429a-ab75-e02c9ad7c97d
2021-11-24T22:20:49.773+0800 DEBUG remote/client.go:169 Request method: POST, path: /api/fleet/agents/282120f3-ddba-450f-8661-a04c4ea6c731/checkin
root@eleong-test-endpoint:/opt/Elastic/Agent#
Also an update, I've even imported the ca cert into the Linux client machine as suggested in another topic, but no luck. I used "update-ca-certificates" command in Ubuntu and saw the certificate updated successfully in /etc/ssl/certs folder.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.