I set up an elastic stack using elastic 8.12. It was working fine and fleet was working as expected. One my elasticsearch server unfortunately died. There were no back ups...
I recreated the entire Elastic, Kibana, and Fleet stack following the guide I had followed previously:
This did work for me previously. I however, run into errors where I get certificate x509 errors from the fleet server and the integrations installed on it despite the server showing up in the Kibana console and reading healthy.
I am able to curl from the fleet server to elasticsearch, the kibana server to elasticsearch and elasticsearch to elasticsearch using the certificates I generated using /usr/share/elasticsearch/bin/elasticsearch-certutil tool.
The curl command looks like this: curl -X GET -u elastic:<password.you.saved.from.last.step> https://<your.ip.or.fqdn.here>:9200/ --cacert /etc/elasticsearch/certs/ca/ca.crt
I get:
"name" : "my-name",
"cluster_name" : "my-cluster-name",
"cluster_uuid" : "my-uuid",
"version" : {
"number" : "8.14.0",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "8d96bbe3bf5fed931f3119733895458eab75dca9",
"build_date" : "2024-06-03T10:05:49.073003402Z",
"build_snapshot" : false,
"lucene_version" : "9.10.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
} ```
on all servers.
I have double checked to confirm that all the ca.crt files I have copied to my fleet and kibana servers are the same and they match. I have confirmed all certs in general are formatted correctly and match what was created using the elasticsearch-certutil.
I can install integrations to the deployed fleetserver and I can see them use the api keys I give them an example is infoblox. I have been able to set up the integration infoblox-ddi, I see the token get used in the console, and when I look at /opt/Elastic/Agent/data/elastic..../logs/httpjson I can see the logs, I can also see in the preceding logs folder the logs for the infoblox integration are collected. But I get the certificate x509 error and the data never makes it to elasticsearch.
I have confirmed that all firewall ports for ports 5601, 8220, and 9200 are open, I have confirmed that 443 is working and healthy.
I have restarted all services and hosts, as well.
I have configured my system integration to collect metrics and logs as well and it does not. My CPU and Memory indcators don't show data from the 5 minute windows of CPU and memory usage of the fleet server and I get, You may not have permissions to see this data, but I am use the elastic super user to test and I still don't seem to have the permissions. I have also confirmed I have super user privileges.
I have tried different elastic stack versions (making sure all kibana, elastic, and fleet are running the same) I did not experience this issue on 8.12 but I am on 8.13.2, .3, .4 and now 8.14.0.
The 509 errors look like this:
```{"log.level":"error","@timestamp":"2024-06-05T20:58:24.009Z","message":"Error dialing x509: certificate signed by unknown authority","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"address":"my_ip:9200","ecs.version":"1.6.0","log.logger":"esclientleg","log.origin":{"file.line":38,"file.name":"transport/logging.go","function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func2"},"service.name":"filebeat","network":"tcp","ecs.version":"1.6.0"}```
```{"log.level":"error","@timestamp":"2024-06-05T20:58:24.003Z","message":"Failed to connect to backoff(elasticsearch(https://my_ip:9200)): Get \"https://my_ip:9200\": x509: certificate signed by unknown authority","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"ecs.version":"1.6.0"}```