Struggling with Elastic Agent Log Forwarding in ELK Stack Setup

[pok_lehbim]

Hello everyone,

I'm in the process of setting up an ELK stack for my home lab, and I've hit a brick wall regarding Elastic Agent's ability to send logs. Despite following the setup carefully and ensuring everything connects, I can't seem to get logs from the Fleet Server or Elastic Agents into Elasticsearch/Kibana. Here’s a rundown of my setup and the issues I'm facing:

System Overview

General Setup:

• ELK stack running in Docker containers on the same host.
• Certificates generated via OpenSSL for secure communication.
• Fleet Server and Elastic Agents installed on the same network.

Network Configuration:

• All servers (Elasticsearch, Kibana, Fleet Server) run on the same host IP.
• All containers (Elasticsearch, Kibana) are also in a docker network.

My Installation Process

1. Certificates Setup: I generated certificates using OpenSSL as follows:openssl genrsa -out ca.key 2048 openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt # Repeat (+sign) for Elasticsearch and Fleet Server
2. Docker-Compose Setup: I used the a Docker Compose file for Elasticsearch and Kibana, you can find the full file on my blog at (pindjouf dot xyz slash posts slash troubleshooting) (don't want to spam links).
3. Fleet Server and Agent Installation:

• I installed Fleet Server and Elastic Agents using commands similar to:sudo ./elastic-agent install --url=hxxps[://]x.x.x.x:8220 --fleet-server-es=hxxps[://]x.x.x.x:9200 again the full details are on my website, otherwise this posts would be way too long.

What Works

• Fleet Server shows as Healthy in Kibana.
• Elastic Agents connect to the Fleet Server.
• Kibana is accessible via its web portal.

What Doesn’t Work

• Fleet Server isn’t sending any logs.
• Elastic Agents appear Degraded with multiple components failing.

Example Logs from the Agent:

{"log.level":"error","@timestamp":"2025-01-09T15:42:13.895Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed log-default (STARTING->FAILED): Failed: pid '69668' exited with code '-1'"}

Status Output ( sudo elastic-agent status ):

┌─ fleet
│ └─ status: (HEALTHY) Connected
└─ elastic-agent
├─ status: (DEGRADED) 1 or more components/units in a failed state
├─ log-default
│ ├─ status: (FAILED) Failed: pid '68906' exited with code '-1'

What I've Tried

1. Verified certificates are valid using OpenSSL.
2. Restarted all services and reinstalled agents multiple times.

Request for Help

I suspect there might be an issue with:

• The communication between the Elastic Agent and Fleet Server.
• A missing configuration in Docker/Elasticsearch that affects log forwarding.
• Something subtle I’ve overlooked in the Fleet Server/Elastic Agent configuration.
• Local issue for the agents' processes.

I’ve documented the full process of my setup on my blog at (pindjouf dot xyz slash posts slash troubleshooting) in case further details are needed.

Additional Questions

1. Is there a better way to debug Elastic Agent logs?
2. Should I try removing components like Docker for a simpler setup, or is this a known issue?

Any help or pointers would be greatly appreciated. Thanks in advance!

[stephenb (Stephen Brown)]

Hi @pok_lehbim Welcome to the community... Nice work...

You can tail the logs... I suspect your agent output are failing to connect with Elasticsearch which is where they ship the data.

sudo elastic-agent logs -f

You want to look were filebeat is trying to connect to elasticsearch I suspect it is failing and there should be some error logs... that last log you show is when it gives up and fails

and you can do this

sudo elastic-agent restart
sudo elastic-agent logs -f | grep connect 

Take a look and see what you find... probably connection error...

Other thing I would look at is what is the default output for Elasticsearch for Fleet ... common mistake is not to change that

Kibana -> Fleet -> Settings - Output

My next guess would be certs issue