Metricbeat elasticsearch module and SSL configuration

tl;dr: I'm having trouble getting metricbeat elasticsearch module to work properly after enabling ssl.

Background
I followed the tutorials to get ssl/tls set up on my test cluster. There is one node that is the elasticsearch 'cluster', a different node that is the kibana server, and a few machines that push data in. The setup was fully functional prior to starting to turn the various security components on, then I tackled user roles last weekend, and then SSL/TLS this weekend.

I used the elasticsearch-certutil to make a couple things

elasticsearch-certutil ca
elasticsearch-certutil cert --ca elastic-stack-ca.p12
elasticsearch-certutil cert -name kibana-server -dns [some ips and domain name]
openssl pkcs12 -in elastic-certificates.p12 -cacerts -nokeys -out elasticsearch-ca.pem

Elastic and Kibana are working together fine after moving the files to the right servers and referencing them in the configs (I'll post a separate question about some issues with that though :wink: )

For the beats, I'm doing this test with file, metric, heart, and packetbeats.

In the main config yml for each of those I've added a few lines. Originally I had 'verification_mode: certificate' per the tutorials but that didn't work, I've had to set it all to 'none' for now. The ce.pem has been copied to the same directories that the individual beat configs are in.

setup.kibana:
  host: "https://[kibanadomainname]:5601"
  ssl.enabled: true
  ssl:
    certificate_authorities: ["${path.config}/elasticsearch-ca.pem"]
    verification_mode: "none"
  username: test_all_access
  password: password

output.elasticsearch:
  hosts: ["[elasticdomainname]:9200"]
  protocol: "https"
  username: test_all_access
  password: password
  ssl:
    certificate_authorities: ["${path.config}/elasticsearch-ca.pem"]
    verification_mode: "none"

This is working to get local metricbeat system module data pushed in to elastic.

Problems

The elasticsearch and elasticsearch-xpack modules need to pull stats from the cluster (I expect similar problems with the kibana modules but haven't gotten to them yet).

Editing the module/elasticsearch.yml to have ssl lines such as:

- module: elasticsearch
  metricsets:
    - node
    - node_stats
    - cluster_stats
    - index
    - index_summary
  period: 4m
  hosts: ["https://[elasticdomainname]:9200"]
  username: elastic
  password: password
  ssl.certificate_authorities: ["${path.config}/elasticsearch-ca.pem"]

Results in an error message from metricbeat about "unable to hash config..."

021-05-31T08:57:19.554-0700	ERROR	[reload]	cfgfile/list.go:69	Unable to hash given config: missing field accessing '0.ssl' (source:'/elastic-configs/metricbeat/metricbeat-modules/elasticsearch-custom.yml')

Same error message if I put the same multi-line SSL config blob that I'd put in the main metricbeat ymls (which is what I'd started with, just copy/pasting)

If I remove those lines then I get one about:

2021-05-31T08:30:19.144-0700	INFO	module/wrapper.go:259	Error fetching data for metricset elasticsearch.index_summary: error determining if connected Elasticsearch node is master: error making http request: Get "https://foo:9200/_nodes/_local/nodes": x509: certificate signed by unknown authority

Which is 'info' level but still says it is an error that would cause problems...

Question

So what exactly are the module/elastic[].yml and kibana[].ymls supposed to look like to be able to get monitor data in from them once security is turned on?

Hello,

Try to change ${path.config} to /etc/metricbeat.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.