ECE 2.3
Deployment - Elastic 7.2 & Kibana 7.2
Metricbeat-OSS-7.0.0
I'm running into an issue with metricbeat-oss-7.0.0 where beats that are installed on our Windows servers report to both an Elastic-oss version as well as ECE. Metricbeat-oss-7.0.0 for Linux on the other hand will not not connect to our ECE cluster. The first error message I received was;
Sep 10 11:40:07 <server> metricbeat[13000]: 2019-09-10T11:40:07.463-0400 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(https://xxx.xxx.xxx..gov:9243)): Connection marked as failed because the onConnect callback failed: failed to check for policy name 'metricbeat-7.0.0': (status=403) {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_internal]"}],"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_internal]"},"status":403}: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_internal]"}],"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_internal]"},"status":403}
This was corrected by adding manage_ilm to the cluster privileges of the beats_writer role as described in a few github threads. The error now is;
Sep 10 11:38:56 <server> metricbeat[13000]: 2019-09-10T11:38:56.511-0400 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(https://xxx.xxx.xxx.gov:9243)): Connection marked as failed because the onConnect callback failed: failed to check for alias 'metricbeat-7.0.0': (status=403) : 403 Forbidden:
Xpack monitoring is turn on for both Windows and Linux metricbeats and I've got 14 reporting beats which is the correct amount but no metricbeat data for the 3 linux servers. The only mb data, cpu, memory etc, thats available is from the Windows servers. Its hard to explain but for the time being we have a bit of an odd setup but it's odd that Windows-oss seems to work just fine and Linux ones dont, at least connection wise. The best option may be to try and change the deployment to Elasticsearch 7.0 but maybe theres an easy work around in which I wouldnt have to downgrade.
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["xxx.xxx.xxx.xxx:9200", "xxx.xxx.xxx.xxx:9200", "xxx.xxx.xxx:9200","https://xxx.xxx.xx.gov:9243"]
# Optional protocol and basic auth credentials.
#protocol: "https"
username: "<un>"
password: "<pass"
ssl.certificate_authorities: /etc/metricbeat/root_ca_pub.pem