ECE Xpack Monitoring Connected but not displaying in Kibana

I'm currently trying to setup xpack monitoring on a Metricbeat thats pointing to an ECE cluster. I have been able to get a metricbeat on my desktop with xpack monitoring to connect to an ECE cluster named DevOps. Those settings were implemented on a server that points to a different ECE cluster but when I use the same settings, instance id's changed to point to the correct cluster, I get an error when I go to Monitoring that says

We checked the cluster persistent settings for xpack.monitoring.exporters, and found the reason: Remote exporters indicate a possible misconfiguration: found-user-defined
Using monitoring exporters ship the monitoring data to a remote monitoring cluster is highly recommended as it keeps the integrity of the monitoring data safe no matter what the state of the production cluster. However, as this instance of Kibana could not find any monitoring data, there seems to be a problem with the xpack.monitoring.exporters configuration, or the xpack.monitoring.elasticsearch settings in kibana.yml.

Check that the intended exporters are enabled for sending statistics to the monitoring cluster, and that the monitoring cluster host matches the xpack.monitoring.elasticsearch setting in kibana.yml to see monitoring data in this instance of Kibana.

When I saw this error message in my DevOps cluster I made a few changes as per the documentation on both environmenst. One, edit the cluster settings and add xpack.monitoring.enabled: true then save, two, check the cluster settings GET _cluster/settings (see output below) and if needed run the PUT command below.

PUT /_cluster/settings
{
    "persistent" : {
        "xpack.monitoring.collection.enabled" : "true"
    }
}

part of settings output

"xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true",
          "interval" : "10s"
        },
        "exporters" : {
          "__no-default-local__" : {
            "type" : "local",
            "enabled" : "false"
          },
          "found-user-defined" : {
            "host" : [
              "http://containerhost:9244"
            ],
            "headers" : {
              "x-found-cluster" : "04ea778aa7694ac59a9acf636927dc22"
            },
            "type" : "http",
            "enabled" : "true"

When I restart Metricbeat or Heartbeat I'm see this response in the logs which should indicate that monitoring should populate beats in ES and Kibana

2019-07-19T08:47:05.032-0400	INFO	instance/beat.go:571	Home path: [D:\basefarm\metricbeat-7.0.0-windows-x86_64] Config path: [D:\basefarm\metricbeat-7.0.0-windows-x86_64] Data path: [C:\ProgramData\metricbeat] Logs path: [C:\ProgramData\metricbeat\logs]
2019-07-19T08:47:05.202-0400	INFO	instance/beat.go:579	Beat ID: c902db8c-8019-4557-b7a5-fc4bc7507f77
2019-07-19T08:47:05.222-0400	INFO	[beat]	instance/beat.go:827	Beat info	{"system_info": {"beat": {"path": {"config": "D:\\basefarm\\metricbeat-7.0.0-windows-x86_64", "data": "C:\\ProgramData\\metricbeat", "home": "D:\\basefarm\\metricbeat-7.0.0-windows-x86_64", "logs": "C:\\ProgramData\\metricbeat\\logs"}, "type": "metricbeat", "uuid": "c902db8c-8019-4557-b7a5-fc4bc7507f77"}}}
2019-07-19T08:47:05.289-0400	INFO	[beat]	instance/beat.go:836	Build info	{"system_info": {"build": {"commit": "da192b7d09af1d735cef19ea7816b8b8a5d4a323", "libbeat": "7.0.0", "time": "2019-04-05T22:01:57.000Z", "version": "7.0.0"}}}
2019-07-19T08:47:05.289-0400	INFO	[beat]	instance/beat.go:839	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.11.5"}}}
2019-07-19T08:47:05.294-0400	INFO	[beat]	instance/beat.go:843	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-07-17T04:56:53.16-04:00","name":"xxxxxxxxx","ip":["fe80::7c1a:8fb8:93f9:91a0/64","xxxxxxxxx","::1/128","xxxxxxxxx"],"kernel_version":"xxxxxxxxxs1_release.190703-1816)","mac":["xxxxxxxxx"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2016 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"14393.3085"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"9b022945-0518-4024-a781-957b584eff65"}}}
2019-07-19T08:47:05.299-0400	INFO	[beat]	instance/beat.go:872	Process info	{"system_info": {"process": {"cwd": "C:\\Windows\\system32", "exe": "D:\\basefarm\\metricbeat-7.0.0-windows-x86_64\\metricbeat.exe", "name": "metricbeat.exe", "pid": 798224, "ppid": 624, "start_time": "2019-07-19T08:47:01.448-0400"}}}
2019-07-19T08:47:05.299-0400	INFO	instance/beat.go:280	Setup Beat: metricbeat; Version: 7.0.0
2019-07-19T08:47:05.356-0400	INFO	elasticsearch/client.go:165	Elasticsearch url: https://xxxxxxxxx.ny.gov:443
2019-07-19T08:47:05.383-0400	INFO	[publisher]	pipeline/module.go:97	Beat name: xxxxxxxxx
2019-07-19T08:47:05.385-0400	INFO	elasticsearch/client.go:165	Elasticsearch url: https://xxxxxxxxx.gov:443
2019-07-19T08:47:05.385-0400	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2019-07-19T08:47:05.385-0400	INFO	kibana/client.go:118	Kibana url: https://xxxxxxxxx.gov:443
2019-07-19T08:47:05.442-0400	INFO	[monitoring]	elasticsearch/elasticsearch.go:255	Successfully connected to X-Pack Monitoring endpoint.
2019-07-19T08:47:05.442-0400	INFO	[monitoring]	elasticsearch/elasticsearch.go:269	Start monitoring stats metrics snapshot loop with period 10s.
2019-07-19T08:47:05.442-0400	INFO	[monitoring]	elasticsearch/elasticsearch.go:269	Start monitoring state metrics snapshot loop with period 1m0s.
2019-07-19T08:47:05.968-0400	INFO	kibana/client.go:118	Kibana url: https://xxxxxxxxx.gov:443
2019-07-19T08:47:08.231-0400	INFO	add_cloud_metadata/add_cloud_metadata.go:340	add_cloud_metadata: hosting provider type not detected.
2019-07-19T08:47:16.445-0400	INFO	pipeline/output.go:95	Connecting to backoff(publish(elasticsearch(https://xxxxxxxxx.gov:443)))
2019-07-19T08:47:16.446-0400	INFO	pipeline/output.go:105	Connection to backoff(publish(elasticsearch(https://xxxxxxxxx.gov:443))) established
2019-07-19T08:47:16.464-0400	INFO	elasticsearch/client.go:734	Attempting to connect to Elasticsearch version 7.0.0
2019-07-19T08:47:34.322-0400	INFO	instance/beat.go:741	Kibana dashboards successfully loaded.
2019-07-19T08:47:34.322-0400	INFO	instance/beat.go:391	metricbeat start running.

Seems like the beats are connecting and establishing xpack monitoring but they wont display in Kibana. What am I missing?

I should add the differences between the MB yamls as well for some more info. This is just the parts that are different.

Working MB:

#==================== Elasticsearch template setting ==========================
setup.template.enabled: false
#setup.template.settings:
  #index.number_of_shards: 1
  #index.number_of_replicas: 0
  #index.codec: best_compression
  #_source.enabled: false

setup.ilm.enabled: false

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]
  hosts: ["xxxxxxxxx.gov"]

#============================== Xpack Monitoring ===============================
# Set to true to enable the monitoring reporter.
monitoring.enabled: true

monitoring.elasticsearch:
  hosts: ["https://xxxxxxxxx.gov:443"]
  username: "xxxxxxxxx"
  password: "xxxxxxxxx"

Metricbeat that is connected and sending data but is not showing up in Monitoring

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  index.codec: best_compression
  #_source.enabled: false
setup.ilm.enabled: false

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://xxxxxxxxx.gov:443"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"
  ssl.certificate_authorities: D:\basefarm\ECE\xxxxxxxxx.pem
  username: "xxxxxxxxx"
  password: "xxxxxxxxx"

#============================== Xpack Monitoring ===============================
# Set to true to enable the monitoring reporter.
xpack.monitoring.enabled: true

xpack.monitoring.elasticsearch:
  username: "xxxxxxxxx"
  password: "xxxxxxxxx"

Hi @Ryan_Downey,

Do you see any .monitoring-beats-* indices on https://xxxxxxxxx.gov:443? You can get this list by going to https://xxxxxxxxx.gov:443/_cat/indices

I do not see .monitoring-beats-* in the cluster where I'm having monitoring issues. I do see those indices, listed by date, in the cluster where it is working.

Quick edit. These are metricbeats on windows computes/server going to a RHEL based ECE 2.2 deployment.

Is the ES instance for this one located at localhost:9200? That's the default and I don't see any explicit host information in the configuration for the non-working beat instance:

xpack.monitoring.enabled: true

xpack.monitoring.elasticsearch:
  username: "xxxxxxxxx"
  password: "xxxxxxxxx"

So that was one of my troubleshooting steps but I get an error when I try and add the ES cluster to the yaml.

2019-07-19T09:44:37.171-0400	INFO	instance/beat.go:571	Home path: [D:\basefarm\metricbeat-7.0.0-windows-x86_64] Config path: [D:\basefarm\metricbeat-7.0.0-windows-x86_64] Data path: [C:\ProgramData\metricbeat] Logs path: [C:\ProgramData\metricbeat\logs]
2019-07-19T09:44:37.256-0400	INFO	instance/beat.go:579	Beat ID: 544fd60e-0d92-4022-928f-f9760e5a2332
2019-07-19T09:44:37.266-0400	INFO	[beat]	instance/beat.go:827	Beat info	{"system_info": {"beat": {"path": {"config": "D:\\basefarm\\metricbeat-7.0.0-windows-x86_64", "data": "C:\\ProgramData\\metricbeat", "home": "D:\\basefarm\\metricbeat-7.0.0-windows-x86_64", "logs": "C:\\ProgramData\\metricbeat\\logs"}, "type": "metricbeat", "uuid": "544fd60e-0d92-4022-928f-f9760e5a2332"}}}
2019-07-19T09:44:37.267-0400	INFO	[beat]	instance/beat.go:836	Build info	{"system_info": {"build": {"commit": "da192b7d09af1d735cef19ea7816b8b8a5d4a323", "libbeat": "7.0.0", "time": "2019-04-05T22:01:57.000Z", "version": "7.0.0"}}}
2019-07-19T09:44:37.267-0400	INFO	[beat]	instance/beat.go:839	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.11.5"}}}
2019-07-19T09:44:37.272-0400	INFO	[beat]	instance/beat.go:843	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-07-16T03:25:53.51-04:00","name":"xxxxxxxxx","ip":["fe80::7d35:a071:3d27:aa09/64","xxxxxxxxx","::1/128","xxxxxxxxx"],"kernel_version":"10.0.14393.3085 (rs1_release.190703-1816)","mac":["00:50:56:a6:7e:81"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2016 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"14393.3085"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"8315711d-7fad-4cac-b0e4-b396ca6a773d"}}}
2019-07-19T09:44:37.276-0400	INFO	[beat]	instance/beat.go:872	Process info	{"system_info": {"process": {"cwd": "C:\\Windows\\system32", "exe": "D:\\basefarm\\metricbeat-7.0.0-windows-x86_64\\metricbeat.exe", "name": "metricbeat.exe", "pid": 8748, "ppid": 672, "start_time": "2019-07-19T09:44:36.217-0400"}}}
2019-07-19T09:44:37.277-0400	INFO	instance/beat.go:280	Setup Beat: metricbeat; Version: 7.0.0
2019-07-19T09:44:37.287-0400	INFO	elasticsearch/client.go:165	Elasticsearch url: https://cluster_2.gov:443
2019-07-19T09:44:37.289-0400	INFO	[publisher]	pipeline/module.go:97	Beat name: xxxxxxxxx
2019-07-19T09:44:37.289-0400	INFO	instance/beat.go:370	metricbeat stopped.
2019-07-19T09:44:37.291-0400	ERROR	instance/beat.go:802	Exiting: 'xpack.monitoring.elasticsearch.hosts' and 'output.elasticsearch.hosts' are configured

That version of Metricbeat doesn't support separate hosts configured for the ES output and the ES monitoring output. Starting in 7.2, we support separate configured hosts, but note the configuration keys are different (monitoring.* instead of xpack.monitoring.*)

Ok, now that you mention it this beat was setup as a metricbeat-oss version. Does this version prevent the use of xpack monitoring?

Correct, monitoring requires our default distribution, discussed in the Notes section on this page: https://www.elastic.co/downloads/beats/metricbeat

Cool, the environment that isnt working was setup at a pretty basic level which is why we used OSS and now were trying to connect it with ECE for a much larger implementation. I will start setting up the correct metricbeas on the servers etc to get this up and running. Appreciate the insight and quick response Chris. Thanks for your time and enjoy your day!

Sounds good. Good luck and let us know if you have any other issues!

1 Like

Update for others that may come across this. I was unable to get a non-oss 7.0 metricbeat to work so I upgraded our ECE cluster to 7.2 and then downloaded metricbeat-7.2 onto the appropriate servers. I changed the yaml for 7.2 to have the settings below which threw the same "Exiting: 'xpack.monitoring.elasticsearch.hosts' and 'output.elasticsearch.hosts' are configured" error but somewhere along the way another error "DEPRECATED: xpack.monitoring.* settings are deprecated. Use monitoring.* instead, but set monitoring.elasticsearch.hosts to monitoring cluster hosts." came up. So I changed the settings to the second set below and monitoring started working.

xpack.monitoring.enabled: true

xpack.monitoring.elasticsearch:
  hosts: ["https://xxxxxxxxxx.gov:443"]
  username: "xxxxxxxxx"
  password: "xxxxxxxxx"

Working section with xpack removed.

monitoring.enabled: true

monitoring.elasticsearch:
  hosts: ["https://xxxxxxxxx.gov:443"]
  username: "xxxxxxxxx"
  password: "xxxxxxxxx"

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.