Elastic-Agent - metricbeat x509 certificate signed by unknown authority

[KelvUK]

I've elastic-agent running as a docker container configured as the fleet-server. The fleet-server registers without issue and can receive policy updates. However I'm getting no data from the metricbeat component of the agent and when I inspect the logs at /usr/share/elastic-agent/state/data/logs/default I see a recurring error in the metricbeat-20220514-1.ndjson logfile...

{"log.level":"error","@timestamp":"2022-05-14T18:31:43.458Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://elasticsearch.mydomain:9200)): Get "https://elasticsearch.mydomain:9200": x509: certificate signed by unknown authority","service.name":"metricbeat","ecs.version":"1.6.0"}

My docker configuration is...

docker run -it --rm --name fleet --hostname fleet \
--network blacknet10 --ip xxx.xxx.xxx.xxx \
-v /volume1/docker/elasticsearch/config/certs/ca:/tmp/certs/ca \
-v /volume1/docker/acme:/tmp/certs/mydomain \
-e "FLEET_SERVER_ENABLE=true" \
-e "FLEET_URL=https://fleet.mydomain:8220" \
-e "FLEET_SERVER_ELASTICSEARCH_HOST=https://elasticsearch.mydomain:9200" \
-e "FLEET_SERVER_POLICY_ID=fleet-server-policy" \
-e "FLEET_SERVER_ELASTICSEARCH_CA=/tmp/certs/ca/elasticsearch-ca.pem" \
-e "FLEET_SERVER_SERVICE_TOKEN=mytoken" \
-e "FLEET_SERVER_CERT=/tmp/certs/mydomain/certs/*.mydomain.crt" \
-e "FLEET_SERVER_CERT_KEY=/tmp/certs/mydomain/private/*.mydomain.key" \
docker.elastic.co/beats/elastic-agent:8.2.0

I use the same CA file in my kibana docker configuration without issue and manually using the same CA file with curl works just fine. Would appreicate the help as I suspect either my config is bad or my understanding of how the agent should function is bad?

Thanks.

[KelvUK]

Root cause was due to the Output parameters not being fully set/defined for my Elasticsearch host. These can be manually configured in the "Fleet...Settings...Output...Edit output" section in Kibana which is ultimately what I did to solve my issue.

I couldn't find any environment variables to specify to docker which would provide either the Elasticsearch host url or the CA to use when connecting to the Elasticsearch node which would then pre-populate this detail in the Kibana Fleet Settings.

I did locate this post which clarified that an undocumented environment variable called FLEET_SERVER_ELASTICSEARCH_CA_TRUSTED_FINGERPRINT exists which I believe is the same as the Elasticsearch CA trusted fingerprint parameter showed in the same "Kibana...Settings...Output...Edit output" screen (as below).

My new docker config...

> docker run -d --name elasticagent --hostname elasticagent --restart always --user root \
> --network blacknet10 --ip xxx.xxx.xxx.xxx \
> -v /var/run/docker.sock:/var/run/docker.sock \
> -v /volume1/docker/traefik/logs:/tmp/traefik \
> -v /volume1/docker/elasticsearch/certs:/tmp/certs/ca \
> -e "FLEET_ENROLL=1" \
> -e "FLEET_URL=https://mydomain:8220" \
> -e "FLEET_SERVER_ENABLE=true" \
> -e "FLEET_SERVER_ELASTICSEARCH_HOST=https://mydomain:9200" \
> -e "FLEET_SERVER_SERVICE_TOKEN=mytoaken" \
> -e "FLEET_SERVER_ELASTICSEARCH_CA=/tmp/certs/ca/elasticsearch-ca.pem" \
> -e "FLEET_SERVER_POLICY=fleet-server-policy" \
> docker.elastic.co/beats/elastic-agent:8.2.0

I couldn't get the FLEET_SERVER_ELASTICSEARCH_CA_TRUSTED_FINGERPRINT environment variable to work, I believe I ran into a bug being discussed here.

Lastly it would seem the environement variable FLEET_SERVER_ELASTICSEARCH_CA is necessary for fleet server specifications only and isn't used by the metricbeat component of the agent. As such when you inspect the Elastic-Agent logs at /usr/share/elastic-agent/state/data/logs/default you can see the errors about the x509 cert being signed by an unknown authority. To fix this I had to add the yaml config shown in the above screen snip to pass the ssl.certificate_authorities parameter. You can see the edits you make to the "Fleet...Settings...Output...Edit output" by inspecting the /usr/share/elastic-agent/state/data/state.yml file inside the container.