Metricbeat SSL to Elastic Setup

Elastic Stack Beats
1

Hello,

I'm about to lose my mind and scratch ELK all together due to this problem.
Let me start with explaining the goal.

There are 3 VMs.
2 are public VPS' and 1 is a local VM.
All are Ubuntu 20.04.

I will call them bunker, lockhead and home.

Bunker is the ES Node, meant to receive data from metricbeat installed inside itself, and to receive data from "home" and lockhead.
All 3 VMs are different location and would communicate over public internet.

Bunker is a VPS, and currently has Elasticsearch installed and running.
Elasticsearch has been configured to use SSL for Transport and HTTP layer.

        xpack.security.enabled: true
        xpack.security.transport.ssl.enabled: true
        xpack.security.transport.ssl.verification_mode: certificate
        xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
        xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
        xpack.security.http.ssl.enabled: true
        xpack.security.http.ssl.keystore.path: "http.p12"

Grafana is installed in Bunker and is using the CA PEM to talk to the datasource in Elastic.

Metricbeat is also installed here, but I cannot for the life of me get it to use SSL properly, and as such it is not talking to Elasticsearch.

I have followed too many articles to start listing them here, however ES was configured following these steps;
https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html

Metricbeat however, I dont even know where to start... There are talks about .crts, .keys, .this and .that and literally NO logical article on the world wide web that I have found explains HOW these P12s that comes out of certutil is meant to help me.

I tried following this chap;


But it all went south when he started mentioning logstash-ca.crt which I can find NO steps on in his article. Also the fact that I will not be deploying logstash here on "Bunker".

I have read, googled, pulled my hair out - how can it be so incredibly difficult and non-user friendly to get Metricbeat to use SSL?

I managed to get to a point where I told Metricbeat the certificate authorities was the elasticsearch-p12 as listed above, and the certificate was some dodgy crt that came out of openssl, and that the .key was also something out of certutils.
It just whines over and over about "could not find expected key".

The goal is to have lockhead ship data using metricbeat over to bunker over public internet, hence the need for HTTPS. Metricbeat inside Bunker is trying to ship locally, although now ES is in HTTPS mode it has to ship over HTTPS anyway.

So I beg the community, please help me step by step in what I am meant to do to get Metricbeat to use https - this just seems way more complicated than it should be.

My Metricbeat.yml currently has;

      output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["bunkerip:9200"]

      # Protocol - either `http` (default) or `https`.
       protocol: "https"

      # Authentication credentials - either API key or username/password.
      #api_key: "id:api_key"
      username: "DEDACTED"
      password: "DEDACTED"

    ssl.certificate_authorities: ["/etc/metricbeat/metricbeat-ca.crt"]
    ssl.certificate: "/etc/metricbeat/instance.crt"
    ssl.key: "/etc/metricbeat/instance.pkcs8.key"

But I am sure this is not correct what so ever..

1 Like
2

Ive done it!!

Ive finally done it.. Not sure how but I'll try to recap.

As you know, when setting up TLS you will go through the process of creating 2 sets of .P12 files. The Elastic-Stack-ca.P12 and the Elastic Certificate.P12.

What I did afterwards was to extract the CRT + KEY from the elastic-certificate.p12 using the steps here;

openssl pkcs12 -in filename.pfx -nocerts -out filename.key

openssl pkcs12 -in filename.pfx -clcerts -nokeys -out filename.crt

(adjusting the pfx to p12)
I added the KEY to my Metricbeat yml;

  ssl.key: "/etc/metricbeat/elastic.key" 
  ssl.key_passphrase: "DEDACTED" (I have PW protected PEMs)

I then had to create some .PEM files for Metricbeat, which was using;

sudo openssl pkcs12 -in elastic-certificates.p12 -out elastic-ca.pem -clcert -nokeys -passin pass:DEDACTED
sudo openssl pkcs12 -in elastic-stack-ca.p12 -out elastic-ca.pem -clcert -nokeys -passin pass:DEDACTED

I then had 2 .pem files come out.
I was not sure which was going to be which as such, so I tried;

ssl.certificate_authorities: 
    - /etc/metricbeat/elastic-stack-ca.pem
ssl.certificate: "/etc/metricbeat/elastic-ca.pem"

And at the end my metricbeat.yml was showing,

  ssl.certificate_authorities: 
    - /etc/metricbeat/elastic-stack-ca.pem
  ssl.certificate: "/etc/metricbeat/elastic-ca.pem" 
  ssl.key: "/etc/metricbeat/elastic.key" 
  ssl.key_passphrase: "DEDACTED"

However, Metricbeat now failed to start entirely, as it "could not parse private key".
Did a bit of googling on that too and found this;

It was filebeat, but the same symptoms..
Turns out OpenSSL creates dodgy keys, so I had to convert the key;

openssl pkcs8 -in elastic.key -traditional -out plain.pem
openssl rsa -aes256 -in plain.pem -out elasticnew.key

AS POINTED OUT IN THIS GUIDE DELETE THE PLAIN.PEM AFTER AS ITS ENTIRELY UNENCRYPTED
I then updated the elastic.key used inside Metricbeat.yml to the "elasticnew.key", and started up - Bingo bango, Kibana (Using .PEM) now talks to ES, and ES is clearly receiving data.

I'm leaving this answer here should anyone else get stuck. However this should really be more detailed in some form of guide/tutorial.
I spent 2 days trying to get this sorted as a "no-experience-in-ssl" person.

2 Likes
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.