I've seen a few posts like this, but none of them solved my problem, so:
I created a elastic cluster following this tutorial: https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls-docker.html
I've set the built in passwords, and I can reach elasticsearch
$ curl -k --user elastic:$ELASTIC_PWD https://localhost:9200/
{
"name" : "es01",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "hqrGkTUGR0W2Clsaxp75pQ",
"version" : {
"number" : "7.6.2",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
"build_date" : "2020-03-26T06:34:37.794943Z",
"build_snapshot" : false,
"lucene_version" : "8.4.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
I copied the certificate authority to the host machine (where I will be running metricbeat) as follows:
sudo -s
mkdir -p /etc/pki/root
docker exec kibana cat /usr/share/elasticsearch/config/certificates/ca/ca.crt > /etc/pki/root/ca.pem
I can check the certificate is there:
$ cat /etc/pki/root/ca.pem
-----BEGIN CERTIFICATE-----
<certificate>
-----END CERTIFICATE-----
So far so good, then, I installed metricbeat following this tutorial:
https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-installation.html
And set the following configs at /etc/metricbeat/metricbeat.yml:
setup.kibana:
host: "https://localhost:5601"
ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
output.elasticsearch:
hosts: ["https://localhost:9200"]
username: "metricbeat"
password: "<password>"
ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
The metricbeat user I've created in Kibana following this:
https://www.elastic.co/guide/en/beats/metricbeat/current/feature-roles.html
Basically, I created a role called metricbeat_setup with cluster privileges monitor and manage_ilm and index privilege manage on metricbeat-*, then, I created the metricbeat user with roles metricbeat_setup, kibana_admin, ingest_admin, beats_admin, beats_system
I think my problem might be here, I didn't quite understand the necessary permissions so I just went overkill, anyway, continuing...
I ran the setup sudo metricbeat setup -e and everything went fine, no errors in setup, then I started it with sudo service metricbeat start, but nothing gets to elasticsearch, and when I look at sudo service metricbeat status, I get multiple lines of WARN Cannot index event
I tried to look at the logs in /var/log/metricbeat/ but found nothing useful there, also in elasticsearch logs in docker, no useful information as well.
The metricbeat-* index is created and the shard is allocated:
$ curl -k --user elastic:$ELASTIC_PWD https://localhost:9200/_cat/indices/metricbeat-*?pretty
green open metricbeat-7.7.0-2020.05.25-000001 p_0nuiX1S8SARk9QZK01EA 1 1 0 0 566b 283b
$ curl -k --user elastic:$ELASTIC_PWD https://localhost:9200/_cluster/allocation/explain?pretty
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
}
],
"type" : "illegal_argument_exception",
"reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
},
"status" : 400
}
So, please, what am I missing here? I can't find any relevant information in logs or in posts, I've seen posts here saying this was related to lack of disk space, I have 14G free, so I guess that's not the problem. Also, I know metricbeat can connect to both elasticsearch and kibana, because it was able to create the index and the dashboard, I can open the dashboard in kibana but there's no data.
What am I doing wrong?
Thank you