Metricbeat x509 Certificate error

[artschooldropout]

I've got an Elasticsearch instance running nicely, and I'd like to use metricbeat to monitor system performance (running on the same host).

I'm getting an x509 certificate error when I start metricbeat using metricbeat -e.

Here's my metricbeat config:

metricbeat:
  config:
    modules:
      path: /etc/metricbeat/modules.d/*.yml
      reload:
        enabled: false
output:
  elasticsearch:
    hosts:
    - https://localhost:9200
    password: [redacted]
    ssl:
      ca_trusted_fingerprint: [redacted]
      enabled: true
    username: elastic
path:
  config: /etc/metricbeat
  data: /var/lib/metricbeat
  home: /usr/share/metricbeat
  logs: /var/log/metricbeat
processors:
- add_host_metadata: null
- add_cloud_metadata: null
- add_docker_metadata: null
- add_kubernetes_metadata: null
setup:
  kibana: null
  template:
    settings:
      index:
        codec: best_compression
        number_of_shards: 1

Here's the metricbeat log:

https://gist.github.com/packetuser/f4100f27a940938d789143f1bb5746cd

I'm confused, because the same output:elasticsearch section works fine for Filebeat...

I generated a self-signed cert when I set up the ES instance.

Here's my Elasticsearch config:

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch


xpack.security.enabled: true

xpack.security.enrollment.enabled: true

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["zeek1"]

http.host: 0.0.0.0

I know this is a common issue - I've searched to no avail!

[carly.richmond]

Hi @artschooldropout ,

Which version of Metricbeat and Filebeat are you using? Can you try adding the ssl_certificate_authorities option which looks to be missing from your configuration as per this thread?

As an FYI from version 8 onwards you may need to update your certificates to use Subject Alternative Names over CommonName:

@artschooldropout

Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"metricbeat","ecs.version":"1.6.0

[artschooldropout]

I'm using Metricbeat 8.9.1 and Filebeat 8.9.1. I tried adding the ssl_certificate_authorities option as suggested, but I get the same errors.

Here's what my metricbeat config looks like now:

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: "elastic"
  password: "[redacted]"
  ssl:
    enabled: true
    ca_trusted_fingerprint: "[redacted]"
    certificate_authorities: ["/etc/elasticsearch/certs/http_ca.crt"]

I got the cert path from the installation instructions for Debian systems: here

[artschooldropout]

Ok, I tried changing the /elasticsearch-xpack.yml file to match the metricbeat config file, and the output of metricbeat -e seems to indicate it's up and running now. However, I'm getting a 'Monitoring Request Error' in the GUI: [search_phase_execution_exception Root causes: no_shard_available_action_exception: null: search_phase_execution_exception Root causes: no_shard_available_action_exception: null]: all shards failed

[artschooldropout]

Ok - the issue was that my instance was low on disk space. I freed up space and my Metricbeat is working nicely. Thanks @carly.richmond for your help!