Disable monitoring exporters

Hello,
i need to disable monitoring exporters and set the monitoring to default setting

GET /_cluster/settings
 {
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : "true"
          }
        },
        "collection" : {
          "enabled" : "true"
        },
        "exporters" : {
          "cloud_monitoring" : {
            "host" : "",
            "type" : ""
          }
        }
      }
    }
  },
  "transient" : { }
}

i can't delete my cloud_monitoring exporters :frowning:

thank you

ok i found the solution

set value to null

PUT /_cluster/settings 
{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : null
          }
        },
        "collection" : {
          "enabled" : null
        },
        "exporters" : {
          "cloud_monitoring" : {
            "host" : null,
            "type" : null
          }
        }
      }
    }
  }
}

still not working

[2018-11-27T11:52:14,239][WARN ][o.e.c.s.ClusterApplierService] [2yFYVFj] failed to apply cluster settings
org.elasticsearch.common.settings.SettingsException: missing exporter type for [cloud_monitoring] exporter
        at org.elasticsearch.xpack.monitoring.exporter.Exporters.initExporters(Exporters.java:139) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.Exporters.setExportersSetting(Exporters.java:67) ~[?:?]
        at org.elasticsearch.common.settings.Setting$2.apply(Setting.java:625) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.settings.Setting$2.apply(Setting.java:598) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings$SettingUpdater.lambda$updater$0(AbstractScopedSettings.java:555) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.applySettings(AbstractScopedSettings.java:184) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:458) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:416) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:160) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.5.1.jar:6.5.1]

Have you tried this?

PUT /_cluster/settings 
{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : null
          }
        },
        "collection" : {
          "enabled" : null
        },
        "exporters" : {
          "cloud_monitoring" : null
        }
      }
    }
  }
}

no :frowning: i have a status with 400

{
  "error": {
    "root_cause": [
      {
        "type": "remote_transport_exception",
        "reason": "[2yFYVFj][0.0.0.0:9300][cluster:admin/settings/update]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "persistent setting [xpack.monitoring.exporters.cloud_monitoring], not recognized"
  },
  "status": 400
}

If you run GET /_cluster/settings again, what do you see?

i have

{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : { }
}

but i sitll have error and kibana don't show monitoring after 1 hour ....

##
We couldn't activate monitoring

No monitoring data found. Try setting the time filter to "Last 1 hour" or check if data is available for a different time period.

If data is in your cluster, your monitoring dashboards will show up here.

and i sitll have error

[2018-11-27T15:50:38,057][DEBUG][o.e.a.a.c.s.TransportClusterUpdateSettingsAction] [2yFYVFj] failed to perform [cluster_update_settings]
java.lang.IllegalArgumentException: persistent setting [xpack.monitoring.exporters], not recognized
        at org.elasticsearch.common.settings.AbstractScopedSettings.updateSettings(AbstractScopedSettings.java:717) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.updateDynamicSettings(AbstractScopedSettings.java:663) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.action.admin.cluster.settings.SettingsUpdater.updateSettings(SettingsUpdater.java:86) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.action.admin.cluster.settings.TransportClusterUpdateSettingsAction$1.execute(TransportClusterUpdateSettingsAction.java:183) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:639) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:268) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:198) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:133) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.5.1.jar:6.5.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

Can you post the results of these two queries, please?

GET /
POST .monitoring-es-*/_search
{
  "size": 0,
  "aggs": {
    "type": {
      "terms": {
        "field": "type",
        "size": 10
      },
      "aggs": {
        "cluster_uuid": {
          "terms": {
            "field": "cluster_uuid",
            "size": 10
          }
        }
      }
    }
  }
}

Also, please post your kibana.yml after removing or masking any sensitive information like passwords or keys.

{
  "name" : "qkhOC20",
  "cluster_name" : "ES_Cluster_23",
  "cluster_uuid" : "rGGXv2LLQJa-obXgj7PuPQ",
  "version" : {
    "number" : "6.5.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "8c58350",
    "build_date" : "2018-11-16T02:22:42.182257Z",
    "build_snapshot" : false,
    "lucene_version" : "7.5.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"

the seconde query

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : 0.0,
    "hits" : [ ]
  }
}

my kibana.yml

server.host: "0.0.0.0"
elasticsearch.url: "https://xxxx.net:9200"
elasticsearch.username: "kibana"
elasticsearch.password: "XXXX"
logging.silent: false
logging.verbose: true
xpack.security.authorization.legacyFallback.enabled: true
logging.dest: /var/log/kibana.log

Thanks. It looks like there's no monitoring data at all.

Could you also post the results of elasticsearch.yml from your master Elasticsearch node, with any sensitive information masked?

thank you ! but im not using elasticsearch.yml to configure monitoring
i use the kibana gui to configure Elasticsearch
that's worked but when i export monitoring to cloud (elastic cloud monitoring on aws)
then i prefered to keep my monitoring index in my cluster
so after that i have this problem !

I see. In that case, could you post the results of this query?

GET _cluster/settings?filter_path=*.xpack.monitoring&include_defaults=true

Also, are you seeing any errors in your Elasticsearch master node logs about monitoring collection (apart from the IllegalArgumentException: persistent setting [xpack.monitoring.exporters], not recognized error you posted already)?

1 Like
GET /_cluster/settings?filter_path=*.xpack.monitoring&include_defaults=true
{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : "true"
          }
        },
        "collection" : {
          "enabled" : "true"
        },
        "exporters" : {
          "cloud_monitoring" : {
            "type" : "local"
          }
        }
      }
    }
  },
  "defaults" : {
    "xpack" : {
      "monitoring" : {
        "enabled" : "true",
        "collection" : {
          "cluster" : {
            "stats" : {
              "timeout" : "10s"
            }
          },
          "node" : {
            "stats" : {
              "timeout" : "10s"
            }
          },
          "indices" : [ ],
          "ccr" : {
            "stats" : {
              "timeout" : "10s"
            }
          },
          "index" : {
            "stats" : {
              "timeout" : "10s"
            },
            "recovery" : {
              "active_only" : "false",
              "timeout" : "10s"
            }
          },
          "interval" : "10s",
          "ml" : {
            "job" : {
              "stats" : {
                "timeout" : "10s"
              }
            }
          }
        },
        "history" : {
          "duration" : "168h"
        }
      }
    }
  }

I am fairly sure that the GET /_cluster/settings API is hiding the auth.username and auth.password fields in the response.

Try:

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.exporters.cloud_monitoring": {
      "type": null,
      "host": null,
      "auth": {
        "username": null,
        "password": null
      }
    }
  }
}

As an aside, when you just want to toggle it off, it's much easier to just specify "enabled": false as a sub-field, then null that out (or flip it to true) when you want it on again.

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.exporters.cloud_monitoring.enabled": false
  }
}

Thank you

now how to enable local monitoring ?

i have this message

##
You need to make some adjustments

To run monitoring please perform the following steps

We checked the  `cluster persistent`  settings for  `xpack.monitoring.exporters` , and found the reason:  `Exporters are disabled: cloud_monitoring` .

Using monitoring exporters ship the monitoring data to a remote monitoring cluster is highly recommended as it keeps the integrity of the monitoring data safe no matter what the state of the production cluster. However, as this instance of Kibana could not find any monitoring data, there seems to be a problem with the  `xpack.monitoring.exporters`  configuration, or the  `xpack.monitoring.elasticsearch`  settings in  `kibana.yml` .

Check that the intended exporters are enabled for sending statistics to the monitoring cluster, and that the monitoring cluster host matches the  `xpack.monitoring.elasticsearch`  setting in  `kibana.yml`  to see monitoring data in this instance of Kibana.

now how to enable local monitoring ?

By not defining any exporter, the local exporter is implicitly defined and enabled by default. In your case, you've removed the previously defined exporter(s).

It sounds like someone also, after your previous response, added "xpack.monitoring.exporters.cloud_monitoring.enabled": false to your cluster settings (or elasticsearch.yml). This is what the Monitoring UI is complaining about.

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.exporters.cloud_monitoring.enabled": null
  }
}

The next step will be to re-enable xpack.monitoring.collection.enabled as you nulled that out in your example:

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}

still not working :frowning_face:

image

GET /_cluster/settings
{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : { }
}

and i have .monitoring indices

green open .monitoring-es-6-2018.12.08     vkPWL4ClRYGeS2eOC-RrJg 1 1 34559 0 30.8mb 15.4mb
green open .monitoring-es-6-2018.12.05     dllATBhiRhuQQIkAJPVGaw 1 1 34560 0 30.8mb 15.4mb
green open .monitoring-kibana-6-2018.12.05 1mmOz_SmSiqskqpZ-Ge1Bw 1 1 34554 0 17.8mb  8.9mb
green open .monitoring-kibana-6-2018.12.03 IHeGKghFTryHLII2JOD7tQ 1 1 52997 0 23.7mb 11.8mb
green open .monitoring-kibana-6-2018.12.09 AO9NL_wpQB2Z6nh-aNDTaA 1 1 34553 0 17.8mb  8.9mb
green open .monitoring-es-6-2018.12.06     axFp_JsLQWek111IVaIokw 1 1 34560 0   31mb 15.4mb
green open .monitoring-kibana-6-2018.12.06 L3Y5hjd-QVOQI0ERro36UA 1 1 34552 0 17.5mb  8.7mb
green open .monitoring-es-6-2018.12.04     Rx-SgdtBQfCy0IVZTRIRWg 1 1 34560 0 30.9mb 15.5mb
green open .monitoring-es-6-2018.12.09     AUSV3tD3QbSZ2KB48e-Ehg 1 1 34560 0 30.3mb 15.1mb
green open .monitoring-kibana-6-2018.12.07 V5VrjgA2SyONQORaVS_7Ow 1 1 34554 0 17.8mb  8.9mb
green open .monitoring-es-6-2018.12.03     bamPr41PTbqIJp8f6t5Lkg 1 1 34559 0 31.3mb 15.6mb
green open .monitoring-es-6-2018.12.07     _Rf-pl7gTXmRXWcqRSZTVw 1 1 34559 0 30.9mb 15.4mb
green open .monitoring-es-6-2018.12.10     doKfpZ-8Q8SGCuekLMuwcA 1 1 13412 0 14.9mb  7.4mb
green open .monitoring-kibana-6-2018.12.10 2KKxJBSzR--GZAjOT3-AaQ 1 1 13567 0  7.6mb  3.7mb
green open .monitoring-kibana-6-2018.12.08 tyRc08XERxe30bKuJhNWIA 1 1 34552 0 17.9mb    9mb
green open .monitoring-kibana-6-2018.12.04 gUVhR_EMTiivVzRoBltXsg 1 1 34552 0 17.6mb  8.8mb

This may be something else, namely representative of this issue: https://github.com/elastic/kibana/issues/21013 .

The easiest way to confirm this is:

GET /.monitoring-es-6-*/_search
{
  "size": 0,
  "query": {
    "range": {
      "timestamp": {
        "gte": "now-1h"
      }
    }
  },
  "aggs": {
    "group_by_cluster": {
      "terms": {
        "field": "cluster_uuid"
      },
      "group_by_type": {
        "terms": {
          "field": "type"
        }
      }
    }
  }
}

I'm guessing that you are going to observe node_stats returned, but not cluster_stats. You can read about the causes of missing different types here based on the collector's type. The main page, and thus this screen, is driven by the requirement that we receive the cluster_stats document, which is the formative document for the cluster.

You've clearly enabled data and your indices prove that at least some of the data is making it through. The lack of deletes in the .monitoring-es-6-* also suggests that there are no shards documents, which is another thing that comes from the elected master and implies problems there (e.g., maybe you've configured it at the elasticsearch.yml level to communicate somewhere else).

Hope that helps,
Chris

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.