Elastic Agent - Sucessful Enrollment, but not updating beats configuration

Elastic Stack Beats
1

Elastic Agent Version: 7.17.4
Install method: APT
OS: Ubuntu 18.04
Problem Description:

I've installed Elastic Agent, and successfully enrolled it into the Fleet within my ES cluster.
Within fleet the agent shows "Healthy".

I've attached a policy to the fleet agent through the Kibana UI to start collecting logs and metrics, however Kibana never received any new logs.

sudo elastic-agent status shows:

Status: HEALTHY
Message: (no message)
Applications:
  * filebeat             (CONFIGURING)
                         Updating configuration
  * filebeat_monitoring  (CONFIGURING)
                         Updating configuration

/var/log/elastic-agent/* shows

2022-06-23T12:07:03.970-0400    INFO    stateresolver/stateresolver.go:66       Updating internal state
2022-06-23T12:07:04.042-0400    INFO    [api]   api/server.go:62        Starting stats endpoint
2022-06-23T12:07:04.042-0400    INFO    application/managed_mode.go:291 Agent is starting
2022-06-23T12:07:04.042-0400    INFO    [api]   api/server.go:64        Metrics endpoint listening on: /var/lib/elastic-agent/data/tmp/elastic-agent.sock (configured: unix:///var/lib/elastic-agent/data/tmp/elastic-agent.sock)
2022-06-23T12:07:07.420-0400    INFO    log/reporter.go:40      2022-06-23T12:07:07-04:00 - message: Application: filebeat--7.17.4[8ac44029-f623-4ada-ac0c-ed206e643558]: State changed to CONFIG: Updating configuration - type: 'STATE' - sub_type: 'CONFIG'
2022-06-23T12:07:07.737-0400    INFO    log/reporter.go:40      2022-06-23T12:07:07-04:00 - message: Application: filebeat--7.17.4--36643631373035623733363936343635[8ac44029-f623-4ada-ac0c-ed206e643558]: State changed to CONFIG: Updating configuration - type: 'STATE' - sub_type: 'CONFIG'

Note: debug logging is enabled

If I'm understaning how Elastic Agent works I should see changes populate to
/var/lib/elastic-agent/data/elastic-agent-*/install/filebeat-7.17.4-linux-x76_64/file.yml which seems to be the elastic agent managed filebeat.yml file - But I'm still seeing it points to localhost:9200 (the default configuration)

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

Meanwhile the /var/lib/elastic-agent/data/elastic-agent-*/state.yml has all the correct information for the policy I created including:

action:
  action_id: policy:<IDSTRING>
  action_type: POLICY_CHANGE
  policy:
    agent:
      monitoring:
        enabled: true
        logs: true
        metrics: false
        namespace: default
        use_output: default
    fleet:
      hosts:
      - https://my-fleet-server1:8220
      - https://my-fleet-server2:8220
      - https://my-fleet-server3:8220
    id: <IDSTRING>

If I'm understanding how Elastic Agent works - You set a policy in the fleet UI which gets downloaded to the state.yml of all the elastic agents enrolled in that fleet. Then elastic agent reads the state.yml and pushes appropriate changes to the related beats in my case filebeat.

If I'm incorrect, whats happening here and why is elastic-agent status telling me it's trying to figure filebeat, but showing no progress or debug messages in any logs?

2

For anyone who stumbles upon this and has a related issue -

My problem was that fleet was looking for an SSL certificate in a directory that didn't exist; there were no logs to indicate this was the problem.

After adding the certificate to that directory - elastic-agent status showed the beats were healthy

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.