I'm about to lose my mind and scratch ELK all together due to this problem.
Let me start with explaining the goal.
There are 3 VMs.
2 are public VPS' and 1 is a local VM.
All are Ubuntu 20.04.
I will call them bunker, lockhead and home.
Bunker is the ES Node, meant to receive data from metricbeat installed inside itself, and to receive data from "home" and lockhead.
All 3 VMs are different location and would communicate over public internet.
Bunker is a VPS, and currently has Elasticsearch installed and running.
Elasticsearch has been configured to use SSL for Transport and HTTP layer.
xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: elastic-certificates.p12 xpack.security.http.ssl.enabled: true xpack.security.http.ssl.keystore.path: "http.p12"
Grafana is installed in Bunker and is using the CA PEM to talk to the datasource in Elastic.
Metricbeat is also installed here, but I cannot for the life of me get it to use SSL properly, and as such it is not talking to Elasticsearch.
I have followed too many articles to start listing them here, however ES was configured following these steps;
Metricbeat however, I dont even know where to start... There are talks about .crts, .keys, .this and .that and literally NO logical article on the world wide web that I have found explains HOW these P12s that comes out of certutil is meant to help me.
I tried following this chap;
But it all went south when he started mentioning logstash-ca.crt which I can find NO steps on in his article. Also the fact that I will not be deploying logstash here on "Bunker".
I have read, googled, pulled my hair out - how can it be so incredibly difficult and non-user friendly to get Metricbeat to use SSL?
I managed to get to a point where I told Metricbeat the certificate authorities was the elasticsearch-p12 as listed above, and the certificate was some dodgy crt that came out of openssl, and that the .key was also something out of certutils.
It just whines over and over about "could not find expected key".
The goal is to have lockhead ship data using metricbeat over to bunker over public internet, hence the need for HTTPS. Metricbeat inside Bunker is trying to ship locally, although now ES is in HTTPS mode it has to ship over HTTPS anyway.
So I beg the community, please help me step by step in what I am meant to do to get Metricbeat to use https - this just seems way more complicated than it should be.
My Metricbeat.yml currently has;
output.elasticsearch: # Array of hosts to connect to. hosts: ["bunkerip:9200"] # Protocol - either `http` (default) or `https`. protocol: "https" # Authentication credentials - either API key or username/password. #api_key: "id:api_key" username: "DEDACTED" password: "DEDACTED" ssl.certificate_authorities: ["/etc/metricbeat/metricbeat-ca.crt"] ssl.certificate: "/etc/metricbeat/instance.crt" ssl.key: "/etc/metricbeat/instance.pkcs8.key"
But I am sure this is not correct what so ever..