I have a Fleet of Elastic Agents running on a fresh k8s cluster (running latest ECK). I added a couple integrations via Kibana, like the Kubernetes metrics. But the Kubernetes Dashboards are all empty, and nothing shows up in Metrics.
The agents all report healthy in Fleet Management. I can spot no clues from the logs on fleet-server nor elastic-agent pods. Any help would be greatly appreciated.
Here are some more details of my debugging journey:
Initially, I saw that the official quickstart has a couple of typos, but the one specific to this issue was:
https://fleet-server-agent-http.default.svc:8220 should be
https://fleet-server-quickstart-agent-http.default.svc:8220 After fixing and applying this, the elastic-agent logs still kept saying:
2021-08-25T04:54:11.727Z WARN [transport] transport/tcp.go:52 DNS lookup failure "fleet-server-agent-http.default.svc": lookup fleet-server-agent-http.default.svc on 10.100.0.10:53: no such host 2021-08-25T04:54:11.728Z ERROR fleet/fleet_gateway.go:180 failed to dispatch actions, error: fail to communicate with updated API client hosts: Get "https://fleet-server-agent-http.default.svc:8220/api/status?": lookup fleet-server-agent-http.default.svc on 10.100.0.10:53: no such host
I forced restart the elastic agents and kibana, but that didn't work. I had to go into the kibana UI and find a random Fleet setting that still said
fleet-server-agent... which I had to change to
All of that takes me to my current situation.