Failed to verify minimum version [5.0.0-beta1] on the [xpack.monitoring.exporters.id1] monitoring cluster

Hi,

I want to send the elasticsearch monitoring data to a separate monitoring cluster. Both the cluster have Elasticsearch 5.2.0. In the actual cluster, in elasticsearch.yml I added the following configuration. Note: Password is changed for security reasons.

xpack.monitoring.exporters:
id1:
type: http
host: 10.2.29.182:9200
auth:
username: elastic
password: xyz

GET request to monitoring cluster is working using the same credentails. Even kibana access to monitoring cluster is working with the credentials, where as when monitoring data is being sent to monitoring cluster, HTTP Code 401
, Unauthorized is being returned. See below for more details.

curl -v -u elastic:xyz -XGET 'http://10.2.29.182:9200/_xpack/security/user/remote_monitor?pretty'

  • About to connect() to 10.2.29.182 port 9200 (#0)
  • Trying 10.2.29.182...
  • Connected to 10.2.29.182 (10.2.29.182) port 9200 (#0)
  • Server auth using Basic with user 'elastic'

GET /_xpack/security/user/remote_monitor?pretty HTTP/1.1
User-Agent: curl/7.29.0
Host: 10.2.29.182:9200
Accept: /

< HTTP/1.1 200 OK
< Content-Type: application/json; charset=UTF-8
< Content-Length: 242
< Connection: keep-alive
<
{
"remote_monitor" : {
"username" : "remote_monitor",
"roles" : [
"remote_monitoring_agent"
],
"full_name" : "Internal Agent For Remote Monitoring",
"email" : null,
"metadata" : { },
"enabled" : true
}
}

This error is being printed every 10 sec in the main cluster which tried to send logs.

[2017-07-31T12:29:21,698][INFO ][o.e.x.m.e.Exporters ] [2YC2PWt] skipping exporter [id1] as it is not ready yet
[2017-07-31T12:29:31,742][ERROR][o.e.x.m.e.h.VersionHttpResource] failed to verify minimum version [5.0.0-beta1] on the [xpack.monitoring.exporters.id1] monitoring cluster
org.elasticsearch.client.ResponseException: GET http://10.2.29.182:9200/?filter_path=version.number: HTTP/1.1 401 Unauthorized
{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}

Can anyone point out what is going wrong here?

Update: 5.5.0 returned 404 for several URL's. Some of them are below. Ofcourse, I didn't configure any templates since default monitoring is also not working for me. Is it mandatory to configure templates as mentioned in https://www.elastic.co/guide/en/x-pack/current/configuring-monitoring.html
Host: 10.2.29.182:9200
Request: GET HTTP/1.1 (252 B)
URI: /_ingest/pipeline/xpack_monitoring_6 ? filter_path=.version
/_template/.monitoring-alerts-2 ? filter_path=
.version
/_template/.monitoring-kibana ? filter_path=*.version
Regards,
Sindhura

What does _cat/nodes?v&h=id,ip,v,m show for each of the clusters?

From Main Cluster,

id ip v m
A3pP 192.168.133.9 5.2.0 -
2YC2 192.168.137.6 5.2.0 -
R1qx 192.168.149.3 5.2.0 -
6y5K 192.168.156.5 5.2.0 -
cR03 192.168.168.5 5.2.0 -
dDIp 192.168.169.15 5.2.0 -
Ozi3 192.168.156.4 5.2.0 -
1BrV 192.168.133.8 5.2.0 *
1cGA 192.168.137.4 5.2.0 -
80Wq 192.168.133.5 5.2.0 -
We-5 192.168.137.5 5.2.0 -

From Monitoring cluster,
id ip v m
eCI6 192.168.135.3 5.2.0 -
yiRC 192.168.133.3 5.2.0 -
vC4n 192.168.174.2 5.2.0 -
ZE9T 192.168.151.3 5.2.0 *

From monitoring cluster, the indices from today are
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .monitoring-es-6-2017.08.01 FSZMyiNNQv66G9Ekl_HEzg 1 1 154942 799 279.6mb 140mb
green open .monitoring-kibana-6-2017.08.01 x3PCPPZgSYypMqDpwZDjqA 1 1 11106 0 9.6mb 4.7mb

monitoring index has been created and there is some data, but Monitoring page says No Monitoring Data Found and the loadbalancer which we have in front of monitoring cluster shows GTTP 401 error code for requests (atleast 10 times in 5 minutes)
Request: POST HTTP/1.1 (263.1 KB) application/json; charset=UTF-8
URI: /_bulk ? pipeline=xpack_monitoring_2&filter_path=errors%2Citems.*.error

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.