We are rebuilding our cluster using 8.14.1, and decided to change our configuration to match the latest and greatest. We have fleet servers and agents, and all our certificates are generated using our in-house CA - we DO NOT use any elasticsearch generated CA or certificates.
After much trial and error we have got our cluster going using encryption and the fleet/kibana/agent communication seems to work ok.
However, we are not seeing any metric data flowing from any of our agents, and we think it is because we are seeing error messages like this:
{"log.level":"error","@timestamp":"2024-06-19T05:12:16.600Z","message":"Error fetching data for metricset http.json: error making http request: Get \"https://localhost:5601/api/task_manager/metrics\": x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-7ec42e06-a59d-4e52-afe3-9528c2b46e62","type":"http/metrics"},"log":{"source":"http/metrics-7ec42e06-a59d-4e52-afe3-9528c2b46e62"},"log.origin":{"file.line":256,"file.name":"module/wrapper.go","function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
This appears to be an error regarding communications from the Agent process to the local metrics provider (kibana in this case, but we see the same from our elasticsearch and fleet hosts).
Thinking this through, it seems that the process trying to do the metrics connection is not trusting the certificate presented by kibana, because it cannot find the CA certificates. This appears valid, because there were no CA certificates passed to the agent install.
We have seen examples in the Elastic docs where the agent install command line has the option
--certificate-authorities=/path/to/ca.crt
but this seems to imply that the root/issuing CA certificate has already been installed on the agent host somewhere. What if we want to install the agent without explicitly having to previously install the CA certificate?
Is there a work around here?
Should this be an enhancement, so that people can install everything the agent needs at once without having to have installed the CA certificate first?