Metricbeat OSS connects for Windows servers not for Linux

Elastic Stack Beats
1

ECE 2.3
Deployment - Elastic 7.2 & Kibana 7.2
Metricbeat-OSS-7.0.0

I'm running into an issue with metricbeat-oss-7.0.0 where beats that are installed on our Windows servers report to both an Elastic-oss version as well as ECE. Metricbeat-oss-7.0.0 for Linux on the other hand will not not connect to our ECE cluster. The first error message I received was;

Sep 10 11:40:07 <server> metricbeat[13000]: 2019-09-10T11:40:07.463-0400        ERROR        pipeline/output.go:100        Failed to connect to backoff(elasticsearch(https://xxx.xxx.xxx..gov:9243)): Connection marked as failed because the onConnect callback failed: failed to check for policy name 'metricbeat-7.0.0': (status=403) {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_internal]"}],"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_internal]"},"status":403}: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_internal]"}],"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_internal]"},"status":403}

This was corrected by adding manage_ilm to the cluster privileges of the beats_writer role as described in a few github threads. The error now is;

Sep 10 11:38:56 <server> metricbeat[13000]: 2019-09-10T11:38:56.511-0400        ERROR        pipeline/output.go:100        Failed to connect to backoff(elasticsearch(https://xxx.xxx.xxx.gov:9243)): Connection marked as failed because the onConnect callback failed: failed to check for alias 'metricbeat-7.0.0': (status=403) : 403 Forbidden:

Xpack monitoring is turn on for both Windows and Linux metricbeats and I've got 14 reporting beats which is the correct amount but no metricbeat data for the 3 linux servers. The only mb data, cpu, memory etc, thats available is from the Windows servers. Its hard to explain but for the time being we have a bit of an odd setup but it's odd that Windows-oss seems to work just fine and Linux ones dont, at least connection wise. The best option may be to try and change the deployment to Elasticsearch 7.0 but maybe theres an easy work around in which I wouldnt have to downgrade.

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["xxx.xxx.xxx.xxx:9200", "xxx.xxx.xxx.xxx:9200", "xxx.xxx.xxx:9200","https://xxx.xxx.xx.gov:9243"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "<un>"
  password: "<pass"
  ssl.certificate_authorities: /etc/metricbeat/root_ca_pub.pem
2

This is interesting. Out of curiosity, can you try and disable ILM on the linux boxes?

1 Like
3

@Alex_Kristiansen your assumption was correct! I set the Linux boxes kibana yaml to setup.ilm.enabled: false and data is now populating into our ECE cluster which is nice to see. Thanks for the assist. Will mark this one as solved!
Ryan

4

That was @dedemorton's idea, thank her! Glad it worked, though.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.