tl;dr: I'm having trouble getting metricbeat elasticsearch module to work properly after enabling ssl.
Background
I followed the tutorials to get ssl/tls set up on my test cluster. There is one node that is the elasticsearch 'cluster', a different node that is the kibana server, and a few machines that push data in. The setup was fully functional prior to starting to turn the various security components on, then I tackled user roles last weekend, and then SSL/TLS this weekend.
I used the elasticsearch-certutil to make a couple things
elasticsearch-certutil ca
elasticsearch-certutil cert --ca elastic-stack-ca.p12
elasticsearch-certutil cert -name kibana-server -dns [some ips and domain name]
openssl pkcs12 -in elastic-certificates.p12 -cacerts -nokeys -out elasticsearch-ca.pem
Elastic and Kibana are working together fine after moving the files to the right servers and referencing them in the configs (I'll post a separate question about some issues with that though )
For the beats, I'm doing this test with file, metric, heart, and packetbeats.
In the main config yml for each of those I've added a few lines. Originally I had 'verification_mode: certificate' per the tutorials but that didn't work, I've had to set it all to 'none' for now. The ce.pem has been copied to the same directories that the individual beat configs are in.
setup.kibana:
host: "https://[kibanadomainname]:5601"
ssl.enabled: true
ssl:
certificate_authorities: ["${path.config}/elasticsearch-ca.pem"]
verification_mode: "none"
username: test_all_access
password: password
output.elasticsearch:
hosts: ["[elasticdomainname]:9200"]
protocol: "https"
username: test_all_access
password: password
ssl:
certificate_authorities: ["${path.config}/elasticsearch-ca.pem"]
verification_mode: "none"
This is working to get local metricbeat system module data pushed in to elastic.
Problems
The elasticsearch and elasticsearch-xpack modules need to pull stats from the cluster (I expect similar problems with the kibana modules but haven't gotten to them yet).
Editing the module/elasticsearch.yml to have ssl lines such as:
- module: elasticsearch
metricsets:
- node
- node_stats
- cluster_stats
- index
- index_summary
period: 4m
hosts: ["https://[elasticdomainname]:9200"]
username: elastic
password: password
ssl.certificate_authorities: ["${path.config}/elasticsearch-ca.pem"]
Results in an error message from metricbeat about "unable to hash config..."
021-05-31T08:57:19.554-0700 ERROR [reload] cfgfile/list.go:69 Unable to hash given config: missing field accessing '0.ssl' (source:'/elastic-configs/metricbeat/metricbeat-modules/elasticsearch-custom.yml')
Same error message if I put the same multi-line SSL config blob that I'd put in the main metricbeat ymls (which is what I'd started with, just copy/pasting)
If I remove those lines then I get one about:
2021-05-31T08:30:19.144-0700 INFO module/wrapper.go:259 Error fetching data for metricset elasticsearch.index_summary: error determining if connected Elasticsearch node is master: error making http request: Get "https://foo:9200/_nodes/_local/nodes": x509: certificate signed by unknown authority
Which is 'info' level but still says it is an error that would cause problems...
Question
So what exactly are the module/elastic[].yml and kibana[].ymls supposed to look like to be able to get monitor data in from them once security is turned on?