Not receiving logs from Elastic Agent

eleong · November 20, 2021, 3:14pm

Hi,

I cant seem to receive any logs from elastic agent installed on either Windows or Ubuntu machine. I've searched through the documentations to the best of my abilities but I'm afraid that I'm still too new to Elastic.

My current setup is a simple lab - just a single VM that host Elasticsearch, kibana, logstash, filebeat, and also fleet server. The Elastic agent is installed in a different Windows and Ubuntu VM.

Here's what I've tried/found out so far:

Doing tcpdump on Elastic server, I can see that the Windows and Linux machines are sending something to 8220 and 9200. The server always replies with a RST packet when receiving on 9200.
Both of the Elastic Agents can register perfectly and the status is shown as "Healthy".
I'm using self-signed certificate and already using "--insecure" option when registering.
I can receive logs from agent installed in fleet server.

Here's my Elasticsearch.yml file:

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: abc.ddns.net
xpack.security.enabled: true
discovery.type: single-node
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: optional
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: http.p12
xpack.security.http.ssl.truststore.path: http.p12
xpack.security.http.ssl.verification_mode: certificate
xpack.security.authc.api_key.enabled: true

Here's my kibana.yml file:

elasticsearch.hosts: "https://abc.ddns.net:9200"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/instance.crt
server.ssl.key: /etc/kibana/instance.key
elasticsearch.ssl.certificateAuthorities: "/etc/kibana/elasticsearch-ca.pem"
elasticsearch.ssl.verificationMode: none
xpack.security.enabled: true
xpack.encryptedSavedObjects.encryptionKey: <KEY>
xpack.reporting.encryptionKey: <KEY>
xpack.security.encryptionKey: <KEY>

Here's the output when I do "./elastic-agent inspect" on the Ubuntu client machine:

gent:
  monitoring:
    enabled: true
    logs: true
    metrics: true
    namespace: default
    use_output: default
fleet:
  hosts:
  - https://abc.ddns.net:8220
id: 8b9b1f50-49ff-11ec-bf25-d90e051c15d3
inputs:
- data_stream:
    namespace: default
  id: b928bce6-7ebf-422f-8ba5-7e00c1fec838
  meta:
    package:
      name: system
      version: 1.6.2
  name: system-1
  revision: 1
  streams: 
<CROPPED -- Commands too long>
outputs:
  default:
    api_key: <KEY>
    hosts:
    - https://abc.ddns.net:9200
    protocol: https
    ssl:
      verification_mode: none
    type: elasticsearch
revision: 9

Could you kindly help me?

kvch · November 23, 2021, 3:21pm

Could you get the logs of Elastic Agent from the machine it is running on?

eleong · November 24, 2021, 2:13pm

Hi Noemi,

Here's the logs (I think I got the correct log file?)

root@eleong-test-endpoint:/opt/Elastic/Agent# tail -80 elastic-agent-20211124221553 
2021-11-24T22:15:57.325+0800    DEBUG   status/reporter.go:200  'operator-default-bafebdad' has status 'online'
2021-11-24T22:15:57.325+0800    DEBUG   status/reporter.go:212  'filebeat--7.15.1-36361854' has status 'online'
2021-11-24T22:15:57.325+0800    DEBUG   status/reporter.go:212  'metricbeat--7.15.1-3d432d01' has status 'online'
2021-11-24T22:15:57.325+0800    DEBUG   status/reporter.go:212  'filebeat--7.15.1--36643631373035623733363936343635-c2b62aed' has status 'online'
2021-11-24T22:15:57.325+0800    DEBUG   status/reporter.go:212  'metricbeat--7.15.1--36643631373035623733363936343635-d81e9a56' has status 'online'
2021-11-24T22:15:57.325+0800    DEBUG   operation/operator.go:293       operator is looking for filebeat--7.15.1 in app collection: map[filebeat--7.15.1:0xc0000d4500 filebeat--7.15.1--36643631373035623733363936343635:0xc0000d4640 metricbeat--7.15.1:0xc0000d4140 metricbeat--7.15.1--36643631373035623733363936343635:0xc0001d8dc0]
2021-11-24T22:15:57.325+0800    DEBUG   operation/operation_fetch.go:61 binary 'filebeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/filebeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:57.325+0800    DEBUG   operation/operator.go:273       running operation 'retryable block: operation-fetch operation-verify' for filebeat.7.15.1
2021-11-24T22:15:57.325+0800    DEBUG   operation/operation_fetch.go:61 binary 'filebeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/filebeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:57.326+0800    DEBUG   operation/operation_retryable.go:83     running operation 'operation-verify' of the block 'retryable block: operation-fetch operation-verify'
2021-11-24T22:15:57.542+0800    DEBUG   status/reporter.go:200  'capabilities-e3c763c7' has status 'online'
2021-11-24T22:15:57.543+0800    DEBUG   status/reporter.go:200  'operator-default-bafebdad' has status 'online'
2021-11-24T22:15:57.543+0800    DEBUG   status/reporter.go:212  'filebeat--7.15.1-36361854' has status 'online'
2021-11-24T22:15:57.543+0800    DEBUG   status/reporter.go:212  'metricbeat--7.15.1-3d432d01' has status 'online'
2021-11-24T22:15:57.543+0800    DEBUG   status/reporter.go:212  'filebeat--7.15.1--36643631373035623733363936343635-c2b62aed' has status 'online'
2021-11-24T22:15:57.543+0800    DEBUG   status/reporter.go:212  'metricbeat--7.15.1--36643631373035623733363936343635-d81e9a56' has status 'online'
2021-11-24T22:15:57.543+0800    INFO    log/reporter.go:40      2021-11-24T22:15:57+08:00 - message: Application: filebeat--7.15.1--36643631373035623733363936343635[282120f3-ddba-450f-8661-a04c4ea6c731]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
2021-11-24T22:15:57.609+0800    INFO    [api]   api/server.go:62        Starting stats endpoint
2021-11-24T22:15:57.610+0800    INFO    application/managed_mode.go:291 Agent is starting
2021-11-24T22:15:57.610+0800    INFO    [api]   api/server.go:64        Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
2021-11-24T22:15:57.904+0800    INFO    operation/operator.go:269       operation 'operation-install' skipped for filebeat.7.15.1
2021-11-24T22:15:57.904+0800    INFO    operation/operator.go:269       operation 'operation-start' skipped for filebeat.7.15.1
2021-11-24T22:15:57.904+0800    DEBUG   operation/operator.go:273       running operation 'operation-config' for filebeat.7.15.1
2021-11-24T22:15:57.907+0800    DEBUG   operation/operator.go:293       operator is looking for metricbeat--7.15.1 in app collection: map[filebeat--7.15.1:0xc0000d4500 filebeat--7.15.1--36643631373035623733363936343635:0xc0000d4640 metricbeat--7.15.1:0xc0000d4140 metricbeat--7.15.1--36643631373035623733363936343635:0xc0001d8dc0]
2021-11-24T22:15:57.907+0800    DEBUG   operation/operation_fetch.go:61 binary 'metricbeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/metricbeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:57.907+0800    DEBUG   operation/operator.go:273       running operation 'retryable block: operation-fetch operation-verify' for metricbeat.7.15.1
2021-11-24T22:15:57.907+0800    DEBUG   operation/operation_fetch.go:61 binary 'metricbeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/metricbeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:57.907+0800    DEBUG   operation/operation_retryable.go:83     running operation 'operation-verify' of the block 'retryable block: operation-fetch operation-verify'
2021-11-24T22:15:58.032+0800    DEBUG   fleet/fleet_gateway.go:160      FleetGateway calling Checkin API
2021-11-24T22:15:58.032+0800    DEBUG   fleet/fleet_gateway.go:202      Checking started
2021-11-24T22:15:58.134+0800    DEBUG   fleet/fleet_gateway.go:233      using previously saved ack token: 842b5f13-1993-429a-ab75-e02c9ad7c97d
2021-11-24T22:15:58.134+0800    DEBUG   remote/client.go:169    Request method: POST, path: /api/fleet/agents/282120f3-ddba-450f-8661-a04c4ea6c731/checkin
2021-11-24T22:15:58.357+0800    INFO    operation/operator.go:269       operation 'operation-install' skipped for metricbeat.7.15.1
2021-11-24T22:15:58.357+0800    INFO    operation/operator.go:269       operation 'operation-start' skipped for metricbeat.7.15.1
2021-11-24T22:15:58.357+0800    DEBUG   operation/operator.go:273       running operation 'operation-config' for metricbeat.7.15.1
2021-11-24T22:15:58.387+0800    DEBUG   operation/operator.go:293       operator is looking for filebeat--7.15.1--36643631373035623733363936343635 in app collection: map[filebeat--7.15.1:0xc0000d4500 filebeat--7.15.1--36643631373035623733363936343635:0xc0000d4640 metricbeat--7.15.1:0xc0000d4140 metricbeat--7.15.1--36643631373035623733363936343635:0xc0001d8dc0]
2021-11-24T22:15:58.387+0800    DEBUG   operation/operation_fetch.go:61 binary 'filebeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/filebeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:58.387+0800    DEBUG   operation/operator.go:273       running operation 'retryable block: operation-fetch operation-verify' for filebeat.7.15.1
2021-11-24T22:15:58.387+0800    DEBUG   operation/operation_fetch.go:61 binary 'filebeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/filebeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:58.389+0800    DEBUG   operation/operation_retryable.go:83     running operation 'operation-verify' of the block 'retryable block: operation-fetch operation-verify'
2021-11-24T22:15:58.645+0800    INFO    operation/operator.go:269       operation 'operation-install' skipped for filebeat.7.15.1
2021-11-24T22:15:58.645+0800    INFO    operation/operator.go:269       operation 'operation-start' skipped for filebeat.7.15.1
2021-11-24T22:15:58.645+0800    DEBUG   operation/operator.go:273       running operation 'operation-config' for filebeat.7.15.1
2021-11-24T22:15:58.647+0800    DEBUG   operation/operator.go:293       operator is looking for metricbeat--7.15.1--36643631373035623733363936343635 in app collection: map[filebeat--7.15.1:0xc0000d4500 filebeat--7.15.1--36643631373035623733363936343635:0xc0000d4640 metricbeat--7.15.1:0xc0000d4140 metricbeat--7.15.1--36643631373035623733363936343635:0xc0001d8dc0]
2021-11-24T22:15:58.647+0800    DEBUG   operation/operation_fetch.go:61 binary 'metricbeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/metricbeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:58.647+0800    DEBUG   operation/operator.go:273       running operation 'retryable block: operation-fetch operation-verify' for metricbeat.7.15.1
2021-11-24T22:15:58.647+0800    DEBUG   operation/operation_fetch.go:61 binary 'metricbeat.7.15.1' already exists in /opt/Elastic/Agent/data/elastic-agent-5ae799/downloads/metricbeat-7.15.1-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-11-24T22:15:58.647+0800    DEBUG   operation/operation_retryable.go:83     running operation 'operation-verify' of the block 'retryable block: operation-fetch operation-verify'
2021-11-24T22:15:58.781+0800    DEBUG   status/reporter.go:200  'operator-default-bafebdad' has status 'online'
2021-11-24T22:15:58.781+0800    DEBUG   status/reporter.go:200  'gateway-0521b85b' has status 'online'
2021-11-24T22:15:58.781+0800    DEBUG   status/reporter.go:200  'capabilities-e3c763c7' has status 'online'
2021-11-24T22:15:58.781+0800    DEBUG   status/reporter.go:212  'filebeat--7.15.1-36361854' has status 'online'
2021-11-24T22:15:58.781+0800    DEBUG   status/reporter.go:212  'metricbeat--7.15.1-3d432d01' has status 'online'
2021-11-24T22:15:58.781+0800    DEBUG   status/reporter.go:212  'filebeat--7.15.1--36643631373035623733363936343635-c2b62aed' has status 'online'
2021-11-24T22:15:58.781+0800    DEBUG   status/reporter.go:212  'metricbeat--7.15.1--36643631373035623733363936343635-d81e9a56' has status 'online'
2021-11-24T22:15:58.781+0800    INFO    log/reporter.go:40      2021-11-24T22:15:58+08:00 - message: Application: metricbeat--7.15.1--36643631373035623733363936343635[282120f3-ddba-450f-8661-a04c4ea6c731]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
2021-11-24T22:15:58.853+0800    INFO    operation/operator.go:269       operation 'operation-install' skipped for metricbeat.7.15.1
2021-11-24T22:15:58.853+0800    INFO    operation/operator.go:269       operation 'operation-start' skipped for metricbeat.7.15.1
2021-11-24T22:15:58.853+0800    DEBUG   operation/operator.go:273       running operation 'operation-config' for metricbeat.7.15.1
2021-11-24T22:15:58.856+0800    DEBUG   status/reporter.go:200  'gateway-0521b85b' has status 'online'
2021-11-24T22:15:58.856+0800    DEBUG   status/reporter.go:200  'capabilities-e3c763c7' has status 'online'
2021-11-24T22:15:58.856+0800    DEBUG   status/reporter.go:200  'operator-default-bafebdad' has status 'online'
2021-11-24T22:15:58.856+0800    DEBUG   status/reporter.go:212  'filebeat--7.15.1--36643631373035623733363936343635-c2b62aed' has status 'online'
2021-11-24T22:15:58.856+0800    DEBUG   status/reporter.go:212  'metricbeat--7.15.1--36643631373035623733363936343635-d81e9a56' has status 'online'
2021-11-24T22:15:58.856+0800    DEBUG   status/reporter.go:212  'filebeat--7.15.1-36361854' has status 'online'
2021-11-24T22:15:58.856+0800    DEBUG   status/reporter.go:212  'metricbeat--7.15.1-3d432d01' has status 'online'
2021-11-24T22:15:58.856+0800    INFO    stateresolver/stateresolver.go:66       Updating internal state
2021-11-24T22:20:48.622+0800    DEBUG   dispatcher/dispatcher.go:79     No action to dispatch
2021-11-24T22:20:48.623+0800    DEBUG   fleet/fleet_gateway.go:184      FleetGateway is sleeping, next update in 1s
2021-11-24T22:20:48.623+0800    DEBUG   status/reporter.go:200  'capabilities-e3c763c7' has status 'online'
2021-11-24T22:20:48.623+0800    DEBUG   status/reporter.go:200  'operator-default-bafebdad' has status 'online'
2021-11-24T22:20:48.623+0800    DEBUG   status/reporter.go:200  'gateway-0521b85b' has status 'online'
2021-11-24T22:20:48.623+0800    DEBUG   status/reporter.go:212  'filebeat--7.15.1-36361854' has status 'online'
2021-11-24T22:20:48.623+0800    DEBUG   status/reporter.go:212  'metricbeat--7.15.1-3d432d01' has status 'online'
2021-11-24T22:20:48.623+0800    DEBUG   status/reporter.go:212  'filebeat--7.15.1--36643631373035623733363936343635-c2b62aed' has status 'online'
2021-11-24T22:20:48.623+0800    DEBUG   status/reporter.go:212  'metricbeat--7.15.1--36643631373035623733363936343635-d81e9a56' has status 'online'
2021-11-24T22:20:49.671+0800    DEBUG   fleet/fleet_gateway.go:160      FleetGateway calling Checkin API
2021-11-24T22:20:49.671+0800    DEBUG   fleet/fleet_gateway.go:202      Checking started
2021-11-24T22:20:49.773+0800    DEBUG   fleet/fleet_gateway.go:233      using previously saved ack token: 842b5f13-1993-429a-ab75-e02c9ad7c97d
2021-11-24T22:20:49.773+0800    DEBUG   remote/client.go:169    Request method: POST, path: /api/fleet/agents/282120f3-ddba-450f-8661-a04c4ea6c731/checkin
root@eleong-test-endpoint:/opt/Elastic/Agent#

Also an update, I've even imported the ca cert into the Linux client machine as suggested in another topic, but no luck. I used "update-ca-certificates" command in Ubuntu and saw the certificate updated successfully in /etc/ssl/certs folder.

system · December 22, 2021, 4:14pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.