I'm currently trying to setup xpack monitoring on a Metricbeat thats pointing to an ECE cluster. I have been able to get a metricbeat on my desktop with xpack monitoring to connect to an ECE cluster named DevOps. Those settings were implemented on a server that points to a different ECE cluster but when I use the same settings, instance id's changed to point to the correct cluster, I get an error when I go to Monitoring that says
We checked the cluster persistent settings for xpack.monitoring.exporters, and found the reason: Remote exporters indicate a possible misconfiguration: found-user-defined
Using monitoring exporters ship the monitoring data to a remote monitoring cluster is highly recommended as it keeps the integrity of the monitoring data safe no matter what the state of the production cluster. However, as this instance of Kibana could not find any monitoring data, there seems to be a problem with the xpack.monitoring.exporters configuration, or the xpack.monitoring.elasticsearch settings in kibana.yml.
Check that the intended exporters are enabled for sending statistics to the monitoring cluster, and that the monitoring cluster host matches the xpack.monitoring.elasticsearch setting in kibana.yml to see monitoring data in this instance of Kibana.
When I saw this error message in my DevOps cluster I made a few changes as per the documentation on both environmenst. One, edit the cluster settings and add xpack.monitoring.enabled: true then save, two, check the cluster settings GET _cluster/settings (see output below) and if needed run the PUT command below.
PUT /_cluster/settings
{
"persistent" : {
"xpack.monitoring.collection.enabled" : "true"
}
}
part of settings output
"xpack" : {
"monitoring" : {
"collection" : {
"enabled" : "true",
"interval" : "10s"
},
"exporters" : {
"__no-default-local__" : {
"type" : "local",
"enabled" : "false"
},
"found-user-defined" : {
"host" : [
"http://containerhost:9244"
],
"headers" : {
"x-found-cluster" : "04ea778aa7694ac59a9acf636927dc22"
},
"type" : "http",
"enabled" : "true"
When I restart Metricbeat or Heartbeat I'm see this response in the logs which should indicate that monitoring should populate beats in ES and Kibana
2019-07-19T08:47:05.032-0400 INFO instance/beat.go:571 Home path: [D:\basefarm\metricbeat-7.0.0-windows-x86_64] Config path: [D:\basefarm\metricbeat-7.0.0-windows-x86_64] Data path: [C:\ProgramData\metricbeat] Logs path: [C:\ProgramData\metricbeat\logs]
2019-07-19T08:47:05.202-0400 INFO instance/beat.go:579 Beat ID: c902db8c-8019-4557-b7a5-fc4bc7507f77
2019-07-19T08:47:05.222-0400 INFO [beat] instance/beat.go:827 Beat info {"system_info": {"beat": {"path": {"config": "D:\\basefarm\\metricbeat-7.0.0-windows-x86_64", "data": "C:\\ProgramData\\metricbeat", "home": "D:\\basefarm\\metricbeat-7.0.0-windows-x86_64", "logs": "C:\\ProgramData\\metricbeat\\logs"}, "type": "metricbeat", "uuid": "c902db8c-8019-4557-b7a5-fc4bc7507f77"}}}
2019-07-19T08:47:05.289-0400 INFO [beat] instance/beat.go:836 Build info {"system_info": {"build": {"commit": "da192b7d09af1d735cef19ea7816b8b8a5d4a323", "libbeat": "7.0.0", "time": "2019-04-05T22:01:57.000Z", "version": "7.0.0"}}}
2019-07-19T08:47:05.289-0400 INFO [beat] instance/beat.go:839 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.11.5"}}}
2019-07-19T08:47:05.294-0400 INFO [beat] instance/beat.go:843 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-07-17T04:56:53.16-04:00","name":"xxxxxxxxx","ip":["fe80::7c1a:8fb8:93f9:91a0/64","xxxxxxxxx","::1/128","xxxxxxxxx"],"kernel_version":"xxxxxxxxxs1_release.190703-1816)","mac":["xxxxxxxxx"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2016 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"14393.3085"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"9b022945-0518-4024-a781-957b584eff65"}}}
2019-07-19T08:47:05.299-0400 INFO [beat] instance/beat.go:872 Process info {"system_info": {"process": {"cwd": "C:\\Windows\\system32", "exe": "D:\\basefarm\\metricbeat-7.0.0-windows-x86_64\\metricbeat.exe", "name": "metricbeat.exe", "pid": 798224, "ppid": 624, "start_time": "2019-07-19T08:47:01.448-0400"}}}
2019-07-19T08:47:05.299-0400 INFO instance/beat.go:280 Setup Beat: metricbeat; Version: 7.0.0
2019-07-19T08:47:05.356-0400 INFO elasticsearch/client.go:165 Elasticsearch url: https://xxxxxxxxx.ny.gov:443
2019-07-19T08:47:05.383-0400 INFO [publisher] pipeline/module.go:97 Beat name: xxxxxxxxx
2019-07-19T08:47:05.385-0400 INFO elasticsearch/client.go:165 Elasticsearch url: https://xxxxxxxxx.gov:443
2019-07-19T08:47:05.385-0400 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2019-07-19T08:47:05.385-0400 INFO kibana/client.go:118 Kibana url: https://xxxxxxxxx.gov:443
2019-07-19T08:47:05.442-0400 INFO [monitoring] elasticsearch/elasticsearch.go:255 Successfully connected to X-Pack Monitoring endpoint.
2019-07-19T08:47:05.442-0400 INFO [monitoring] elasticsearch/elasticsearch.go:269 Start monitoring stats metrics snapshot loop with period 10s.
2019-07-19T08:47:05.442-0400 INFO [monitoring] elasticsearch/elasticsearch.go:269 Start monitoring state metrics snapshot loop with period 1m0s.
2019-07-19T08:47:05.968-0400 INFO kibana/client.go:118 Kibana url: https://xxxxxxxxx.gov:443
2019-07-19T08:47:08.231-0400 INFO add_cloud_metadata/add_cloud_metadata.go:340 add_cloud_metadata: hosting provider type not detected.
2019-07-19T08:47:16.445-0400 INFO pipeline/output.go:95 Connecting to backoff(publish(elasticsearch(https://xxxxxxxxx.gov:443)))
2019-07-19T08:47:16.446-0400 INFO pipeline/output.go:105 Connection to backoff(publish(elasticsearch(https://xxxxxxxxx.gov:443))) established
2019-07-19T08:47:16.464-0400 INFO elasticsearch/client.go:734 Attempting to connect to Elasticsearch version 7.0.0
2019-07-19T08:47:34.322-0400 INFO instance/beat.go:741 Kibana dashboards successfully loaded.
2019-07-19T08:47:34.322-0400 INFO instance/beat.go:391 metricbeat start running.
Seems like the beats are connecting and establishing xpack monitoring but they wont display in Kibana. What am I missing?