Disable monitoring exporters

saif · November 27, 2018, 9:53am

Hello,
i need to disable monitoring exporters and set the monitoring to default setting

GET /_cluster/settings
 {
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : "true"
          }
        },
        "collection" : {
          "enabled" : "true"
        },
        "exporters" : {
          "cloud_monitoring" : {
            "host" : "",
            "type" : ""
          }
        }
      }
    }
  },
  "transient" : { }
}

i can't delete my cloud_monitoring exporters

thank you

saif · November 27, 2018, 10:30am

ok i found the solution

set value to null

PUT /_cluster/settings 
{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : null
          }
        },
        "collection" : {
          "enabled" : null
        },
        "exporters" : {
          "cloud_monitoring" : {
            "host" : null,
            "type" : null
          }
        }
      }
    }
  }
}

saif · November 27, 2018, 10:54am

still not working

[2018-11-27T11:52:14,239][WARN ][o.e.c.s.ClusterApplierService] [2yFYVFj] failed to apply cluster settings
org.elasticsearch.common.settings.SettingsException: missing exporter type for [cloud_monitoring] exporter
        at org.elasticsearch.xpack.monitoring.exporter.Exporters.initExporters(Exporters.java:139) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.Exporters.setExportersSetting(Exporters.java:67) ~[?:?]
        at org.elasticsearch.common.settings.Setting$2.apply(Setting.java:625) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.settings.Setting$2.apply(Setting.java:598) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings$SettingUpdater.lambda$updater$0(AbstractScopedSettings.java:555) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.applySettings(AbstractScopedSettings.java:184) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:458) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:416) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:160) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.5.1.jar:6.5.1]

shaunak · November 27, 2018, 11:15am

Have you tried this?

PUT /_cluster/settings 
{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : null
          }
        },
        "collection" : {
          "enabled" : null
        },
        "exporters" : {
          "cloud_monitoring" : null
        }
      }
    }
  }
}

saif · November 27, 2018, 1:17pm

no i have a status with 400

{
  "error": {
    "root_cause": [
      {
        "type": "remote_transport_exception",
        "reason": "[2yFYVFj][0.0.0.0:9300][cluster:admin/settings/update]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "persistent setting [xpack.monitoring.exporters.cloud_monitoring], not recognized"
  },
  "status": 400
}

chrisronline · November 27, 2018, 2:27pm

If you run GET /_cluster/settings again, what do you see?

saif · November 27, 2018, 2:55pm

i have

{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : { }
}

but i sitll have error and kibana don't show monitoring after 1 hour ....

##
We couldn't activate monitoring

No monitoring data found. Try setting the time filter to "Last 1 hour" or check if data is available for a different time period.

If data is in your cluster, your monitoring dashboards will show up here.

and i sitll have error

[2018-11-27T15:50:38,057][DEBUG][o.e.a.a.c.s.TransportClusterUpdateSettingsAction] [2yFYVFj] failed to perform [cluster_update_settings]
java.lang.IllegalArgumentException: persistent setting [xpack.monitoring.exporters], not recognized
        at org.elasticsearch.common.settings.AbstractScopedSettings.updateSettings(AbstractScopedSettings.java:717) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.updateDynamicSettings(AbstractScopedSettings.java:663) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.action.admin.cluster.settings.SettingsUpdater.updateSettings(SettingsUpdater.java:86) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.action.admin.cluster.settings.TransportClusterUpdateSettingsAction$1.execute(TransportClusterUpdateSettingsAction.java:183) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:639) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:268) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:198) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:133) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.5.1.jar:6.5.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

shaunak · November 27, 2018, 3:37pm

Can you post the results of these two queries, please?

GET /

POST .monitoring-es-*/_search
{
  "size": 0,
  "aggs": {
    "type": {
      "terms": {
        "field": "type",
        "size": 10
      },
      "aggs": {
        "cluster_uuid": {
          "terms": {
            "field": "cluster_uuid",
            "size": 10
          }
        }
      }
    }
  }
}

Also, please post your kibana.yml after removing or masking any sensitive information like passwords or keys.

saif · November 27, 2018, 3:50pm

{
  "name" : "qkhOC20",
  "cluster_name" : "ES_Cluster_23",
  "cluster_uuid" : "rGGXv2LLQJa-obXgj7PuPQ",
  "version" : {
    "number" : "6.5.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "8c58350",
    "build_date" : "2018-11-16T02:22:42.182257Z",
    "build_snapshot" : false,
    "lucene_version" : "7.5.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"

the seconde query

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : 0.0,
    "hits" : [ ]
  }
}

my kibana.yml

server.host: "0.0.0.0"
elasticsearch.url: "https://xxxx.net:9200"
elasticsearch.username: "kibana"
elasticsearch.password: "XXXX"
logging.silent: false
logging.verbose: true
xpack.security.authorization.legacyFallback.enabled: true
logging.dest: /var/log/kibana.log

shaunak · November 27, 2018, 3:56pm

Thanks. It looks like there's no monitoring data at all.

Could you also post the results of elasticsearch.yml from your master Elasticsearch node, with any sensitive information masked?

saif · November 27, 2018, 4:03pm

thank you ! but im not using elasticsearch.yml to configure monitoring
i use the kibana gui to configure Elasticsearch
that's worked but when i export monitoring to cloud (elastic cloud monitoring on aws)
then i prefered to keep my monitoring index in my cluster
so after that i have this problem !

shaunak · November 27, 2018, 4:21pm

I see. In that case, could you post the results of this query?

GET _cluster/settings?filter_path=*.xpack.monitoring&include_defaults=true

Also, are you seeing any errors in your Elasticsearch master node logs about monitoring collection (apart from the IllegalArgumentException: persistent setting [xpack.monitoring.exporters], not recognized error you posted already)?

saif · December 3, 2018, 10:45am

GET /_cluster/settings?filter_path=*.xpack.monitoring&include_defaults=true
{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : "true"
          }
        },
        "collection" : {
          "enabled" : "true"
        },
        "exporters" : {
          "cloud_monitoring" : {
            "type" : "local"
          }
        }
      }
    }
  },
  "defaults" : {
    "xpack" : {
      "monitoring" : {
        "enabled" : "true",
        "collection" : {
          "cluster" : {
            "stats" : {
              "timeout" : "10s"
            }
          },
          "node" : {
            "stats" : {
              "timeout" : "10s"
            }
          },
          "indices" : [ ],
          "ccr" : {
            "stats" : {
              "timeout" : "10s"
            }
          },
          "index" : {
            "stats" : {
              "timeout" : "10s"
            },
            "recovery" : {
              "active_only" : "false",
              "timeout" : "10s"
            }
          },
          "interval" : "10s",
          "ml" : {
            "job" : {
              "stats" : {
                "timeout" : "10s"
              }
            }
          }
        },
        "history" : {
          "duration" : "168h"
        }
      }
    }
  }

pickypg · December 5, 2018, 1:27am

I am fairly sure that the GET /_cluster/settings API is hiding the auth.username and auth.password fields in the response.

Try:

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.exporters.cloud_monitoring": {
      "type": null,
      "host": null,
      "auth": {
        "username": null,
        "password": null
      }
    }
  }
}

As an aside, when you just want to toggle it off, it's much easier to just specify "enabled": false as a sub-field, then null that out (or flip it to true) when you want it on again.

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.exporters.cloud_monitoring.enabled": false
  }
}

saif · December 5, 2018, 10:11am

Thank you

now how to enable local monitoring ?

i have this message

##
You need to make some adjustments

To run monitoring please perform the following steps

We checked the  `cluster persistent`  settings for  `xpack.monitoring.exporters` , and found the reason:  `Exporters are disabled: cloud_monitoring` .

Using monitoring exporters ship the monitoring data to a remote monitoring cluster is highly recommended as it keeps the integrity of the monitoring data safe no matter what the state of the production cluster. However, as this instance of Kibana could not find any monitoring data, there seems to be a problem with the  `xpack.monitoring.exporters`  configuration, or the  `xpack.monitoring.elasticsearch`  settings in  `kibana.yml` .

Check that the intended exporters are enabled for sending statistics to the monitoring cluster, and that the monitoring cluster host matches the  `xpack.monitoring.elasticsearch`  setting in  `kibana.yml`  to see monitoring data in this instance of Kibana.

pickypg · December 7, 2018, 10:56pm

now how to enable local monitoring ?

By not defining any exporter, the local exporter is implicitly defined and enabled by default. In your case, you've removed the previously defined exporter(s).

It sounds like someone also, after your previous response, added "xpack.monitoring.exporters.cloud_monitoring.enabled": false to your cluster settings (or elasticsearch.yml). This is what the Monitoring UI is complaining about.

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.exporters.cloud_monitoring.enabled": null
  }
}

The next step will be to re-enable xpack.monitoring.collection.enabled as you nulled that out in your example:

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}

saif · December 10, 2018, 9:26am

still not working

GET /_cluster/settings
{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : { }
}

and i have .monitoring indices

green open .monitoring-es-6-2018.12.08     vkPWL4ClRYGeS2eOC-RrJg 1 1 34559 0 30.8mb 15.4mb
green open .monitoring-es-6-2018.12.05     dllATBhiRhuQQIkAJPVGaw 1 1 34560 0 30.8mb 15.4mb
green open .monitoring-kibana-6-2018.12.05 1mmOz_SmSiqskqpZ-Ge1Bw 1 1 34554 0 17.8mb  8.9mb
green open .monitoring-kibana-6-2018.12.03 IHeGKghFTryHLII2JOD7tQ 1 1 52997 0 23.7mb 11.8mb
green open .monitoring-kibana-6-2018.12.09 AO9NL_wpQB2Z6nh-aNDTaA 1 1 34553 0 17.8mb  8.9mb
green open .monitoring-es-6-2018.12.06     axFp_JsLQWek111IVaIokw 1 1 34560 0   31mb 15.4mb
green open .monitoring-kibana-6-2018.12.06 L3Y5hjd-QVOQI0ERro36UA 1 1 34552 0 17.5mb  8.7mb
green open .monitoring-es-6-2018.12.04     Rx-SgdtBQfCy0IVZTRIRWg 1 1 34560 0 30.9mb 15.5mb
green open .monitoring-es-6-2018.12.09     AUSV3tD3QbSZ2KB48e-Ehg 1 1 34560 0 30.3mb 15.1mb
green open .monitoring-kibana-6-2018.12.07 V5VrjgA2SyONQORaVS_7Ow 1 1 34554 0 17.8mb  8.9mb
green open .monitoring-es-6-2018.12.03     bamPr41PTbqIJp8f6t5Lkg 1 1 34559 0 31.3mb 15.6mb
green open .monitoring-es-6-2018.12.07     _Rf-pl7gTXmRXWcqRSZTVw 1 1 34559 0 30.9mb 15.4mb
green open .monitoring-es-6-2018.12.10     doKfpZ-8Q8SGCuekLMuwcA 1 1 13412 0 14.9mb  7.4mb
green open .monitoring-kibana-6-2018.12.10 2KKxJBSzR--GZAjOT3-AaQ 1 1 13567 0  7.6mb  3.7mb
green open .monitoring-kibana-6-2018.12.08 tyRc08XERxe30bKuJhNWIA 1 1 34552 0 17.9mb    9mb
green open .monitoring-kibana-6-2018.12.04 gUVhR_EMTiivVzRoBltXsg 1 1 34552 0 17.6mb  8.8mb

pickypg · December 10, 2018, 5:33pm

This may be something else, namely representative of this issue: https://github.com/elastic/kibana/issues/21013 .

The easiest way to confirm this is:

GET /.monitoring-es-6-*/_search
{
  "size": 0,
  "query": {
    "range": {
      "timestamp": {
        "gte": "now-1h"
      }
    }
  },
  "aggs": {
    "group_by_cluster": {
      "terms": {
        "field": "cluster_uuid"
      },
      "group_by_type": {
        "terms": {
          "field": "type"
        }
      }
    }
  }
}

I'm guessing that you are going to observe node_stats returned, but not cluster_stats. You can read about the causes of missing different types here based on the collector's type. The main page, and thus this screen, is driven by the requirement that we receive the cluster_stats document, which is the formative document for the cluster.

You've clearly enabled data and your indices prove that at least some of the data is making it through. The lack of deletes in the .monitoring-es-6-* also suggests that there are no shards documents, which is another thing that comes from the elected master and implies problems there (e.g., maybe you've configured it at the elasticsearch.yml level to communicate somewhere else).

Hope that helps,
Chris

system · January 7, 2019, 5:33pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to disable xpack.monitoring? Elasticsearch elastic-stack-monitoring	1	425	November 3, 2020
Monitoring on Elasticsearch Elasticsearch elastic-stack-monitoring	2	441	March 15, 2021
Tried to export monitoring to remote cluster unsuccessfully Elasticsearch elastic-stack-monitoring	2	507	October 14, 2019
Cannot disable "xpack.monitoring.collection.enabled" setting Elasticsearch	2	223	April 21, 2024
Disable stack monitoring in Kibana Kibana elastic-stack-monitoring	3	164	July 31, 2024

Disable monitoring exporters

Related topics