Disable monitoring exporters

monitoring

(Saifeddine Hmissi) #1

Hello,
i need to disable monitoring exporters and set the monitoring to default setting

GET /_cluster/settings
 {
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : "true"
          }
        },
        "collection" : {
          "enabled" : "true"
        },
        "exporters" : {
          "cloud_monitoring" : {
            "host" : "",
            "type" : ""
          }
        }
      }
    }
  },
  "transient" : { }
}

i can't delete my cloud_monitoring exporters :frowning:

thank you


(Saifeddine Hmissi) #2

ok i found the solution

set value to null

PUT /_cluster/settings 
{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : null
          }
        },
        "collection" : {
          "enabled" : null
        },
        "exporters" : {
          "cloud_monitoring" : {
            "host" : null,
            "type" : null
          }
        }
      }
    }
  }
}

(Saifeddine Hmissi) #3

still not working

[2018-11-27T11:52:14,239][WARN ][o.e.c.s.ClusterApplierService] [2yFYVFj] failed to apply cluster settings
org.elasticsearch.common.settings.SettingsException: missing exporter type for [cloud_monitoring] exporter
        at org.elasticsearch.xpack.monitoring.exporter.Exporters.initExporters(Exporters.java:139) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.Exporters.setExportersSetting(Exporters.java:67) ~[?:?]
        at org.elasticsearch.common.settings.Setting$2.apply(Setting.java:625) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.settings.Setting$2.apply(Setting.java:598) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings$SettingUpdater.lambda$updater$0(AbstractScopedSettings.java:555) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.applySettings(AbstractScopedSettings.java:184) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:458) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:416) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:160) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.5.1.jar:6.5.1]

(Shaunak Kashyap) #4

Have you tried this?

PUT /_cluster/settings 
{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : null
          }
        },
        "collection" : {
          "enabled" : null
        },
        "exporters" : {
          "cloud_monitoring" : null
        }
      }
    }
  }
}

(Saifeddine Hmissi) #5

no :frowning: i have a status with 400

{
  "error": {
    "root_cause": [
      {
        "type": "remote_transport_exception",
        "reason": "[2yFYVFj][0.0.0.0:9300][cluster:admin/settings/update]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "persistent setting [xpack.monitoring.exporters.cloud_monitoring], not recognized"
  },
  "status": 400
}

(Chris Roberson) #6

If you run GET /_cluster/settings again, what do you see?


(Saifeddine Hmissi) #7

i have

{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : { }
}

but i sitll have error and kibana don't show monitoring after 1 hour ....

##
We couldn't activate monitoring

No monitoring data found. Try setting the time filter to "Last 1 hour" or check if data is available for a different time period.

If data is in your cluster, your monitoring dashboards will show up here.

and i sitll have error

[2018-11-27T15:50:38,057][DEBUG][o.e.a.a.c.s.TransportClusterUpdateSettingsAction] [2yFYVFj] failed to perform [cluster_update_settings]
java.lang.IllegalArgumentException: persistent setting [xpack.monitoring.exporters], not recognized
        at org.elasticsearch.common.settings.AbstractScopedSettings.updateSettings(AbstractScopedSettings.java:717) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.updateDynamicSettings(AbstractScopedSettings.java:663) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.action.admin.cluster.settings.SettingsUpdater.updateSettings(SettingsUpdater.java:86) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.action.admin.cluster.settings.TransportClusterUpdateSettingsAction$1.execute(TransportClusterUpdateSettingsAction.java:183) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:639) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:268) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:198) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:133) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.5.1.jar:6.5.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

(Shaunak Kashyap) #8

Can you post the results of these two queries, please?

GET /
POST .monitoring-es-*/_search
{
  "size": 0,
  "aggs": {
    "type": {
      "terms": {
        "field": "type",
        "size": 10
      },
      "aggs": {
        "cluster_uuid": {
          "terms": {
            "field": "cluster_uuid",
            "size": 10
          }
        }
      }
    }
  }
}

Also, please post your kibana.yml after removing or masking any sensitive information like passwords or keys.


(Saifeddine Hmissi) #9
{
  "name" : "qkhOC20",
  "cluster_name" : "ES_Cluster_23",
  "cluster_uuid" : "rGGXv2LLQJa-obXgj7PuPQ",
  "version" : {
    "number" : "6.5.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "8c58350",
    "build_date" : "2018-11-16T02:22:42.182257Z",
    "build_snapshot" : false,
    "lucene_version" : "7.5.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"

the seconde query

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : 0.0,
    "hits" : [ ]
  }
}

my kibana.yml

server.host: "0.0.0.0"
elasticsearch.url: "https://xxxx.net:9200"
elasticsearch.username: "kibana"
elasticsearch.password: "XXXX"
logging.silent: false
logging.verbose: true
xpack.security.authorization.legacyFallback.enabled: true
logging.dest: /var/log/kibana.log

(Shaunak Kashyap) #10

Thanks. It looks like there's no monitoring data at all.

Could you also post the results of elasticsearch.yml from your master Elasticsearch node, with any sensitive information masked?


(Saifeddine Hmissi) #11

thank you ! but im not using elasticsearch.yml to configure monitoring
i use the kibana gui to configure Elasticsearch
that's worked but when i export monitoring to cloud (elastic cloud monitoring on aws)
then i prefered to keep my monitoring index in my cluster
so after that i have this problem !


(Shaunak Kashyap) #12

I see. In that case, could you post the results of this query?

GET _cluster/settings?filter_path=*.xpack.monitoring&include_defaults=true

Also, are you seeing any errors in your Elasticsearch master node logs about monitoring collection (apart from the IllegalArgumentException: persistent setting [xpack.monitoring.exporters], not recognized error you posted already)?


(Saifeddine Hmissi) #13
GET /_cluster/settings?filter_path=*.xpack.monitoring&include_defaults=true
{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : "true"
          }
        },
        "collection" : {
          "enabled" : "true"
        },
        "exporters" : {
          "cloud_monitoring" : {
            "type" : "local"
          }
        }
      }
    }
  },
  "defaults" : {
    "xpack" : {
      "monitoring" : {
        "enabled" : "true",
        "collection" : {
          "cluster" : {
            "stats" : {
              "timeout" : "10s"
            }
          },
          "node" : {
            "stats" : {
              "timeout" : "10s"
            }
          },
          "indices" : [ ],
          "ccr" : {
            "stats" : {
              "timeout" : "10s"
            }
          },
          "index" : {
            "stats" : {
              "timeout" : "10s"
            },
            "recovery" : {
              "active_only" : "false",
              "timeout" : "10s"
            }
          },
          "interval" : "10s",
          "ml" : {
            "job" : {
              "stats" : {
                "timeout" : "10s"
              }
            }
          }
        },
        "history" : {
          "duration" : "168h"
        }
      }
    }
  }

(Chris Earle) #14

I am fairly sure that the GET /_cluster/settings API is hiding the auth.username and auth.password fields in the response.

Try:

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.exporters.cloud_monitoring": {
      "type": null,
      "host": null,
      "auth": {
        "username": null,
        "password": null
      }
    }
  }
}

As an aside, when you just want to toggle it off, it's much easier to just specify "enabled": false as a sub-field, then null that out (or flip it to true) when you want it on again.

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.exporters.cloud_monitoring.enabled": false
  }
}

(Saifeddine Hmissi) #15

Thank you

now how to enable local monitoring ?

i have this message

##
You need to make some adjustments

To run monitoring please perform the following steps

We checked the  `cluster persistent`  settings for  `xpack.monitoring.exporters` , and found the reason:  `Exporters are disabled: cloud_monitoring` .

Using monitoring exporters ship the monitoring data to a remote monitoring cluster is highly recommended as it keeps the integrity of the monitoring data safe no matter what the state of the production cluster. However, as this instance of Kibana could not find any monitoring data, there seems to be a problem with the  `xpack.monitoring.exporters`  configuration, or the  `xpack.monitoring.elasticsearch`  settings in  `kibana.yml` .

Check that the intended exporters are enabled for sending statistics to the monitoring cluster, and that the monitoring cluster host matches the  `xpack.monitoring.elasticsearch`  setting in  `kibana.yml`  to see monitoring data in this instance of Kibana.

(Chris Earle) #16

now how to enable local monitoring ?

By not defining any exporter, the local exporter is implicitly defined and enabled by default. In your case, you've removed the previously defined exporter(s).

It sounds like someone also, after your previous response, added "xpack.monitoring.exporters.cloud_monitoring.enabled": false to your cluster settings (or elasticsearch.yml). This is what the Monitoring UI is complaining about.

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.exporters.cloud_monitoring.enabled": null
  }
}

The next step will be to re-enable xpack.monitoring.collection.enabled as you nulled that out in your example:

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}

(Saifeddine Hmissi) #17

still not working :frowning_face:

image

GET /_cluster/settings
{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : { }
}

and i have .monitoring indices

green open .monitoring-es-6-2018.12.08     vkPWL4ClRYGeS2eOC-RrJg 1 1 34559 0 30.8mb 15.4mb
green open .monitoring-es-6-2018.12.05     dllATBhiRhuQQIkAJPVGaw 1 1 34560 0 30.8mb 15.4mb
green open .monitoring-kibana-6-2018.12.05 1mmOz_SmSiqskqpZ-Ge1Bw 1 1 34554 0 17.8mb  8.9mb
green open .monitoring-kibana-6-2018.12.03 IHeGKghFTryHLII2JOD7tQ 1 1 52997 0 23.7mb 11.8mb
green open .monitoring-kibana-6-2018.12.09 AO9NL_wpQB2Z6nh-aNDTaA 1 1 34553 0 17.8mb  8.9mb
green open .monitoring-es-6-2018.12.06     axFp_JsLQWek111IVaIokw 1 1 34560 0   31mb 15.4mb
green open .monitoring-kibana-6-2018.12.06 L3Y5hjd-QVOQI0ERro36UA 1 1 34552 0 17.5mb  8.7mb
green open .monitoring-es-6-2018.12.04     Rx-SgdtBQfCy0IVZTRIRWg 1 1 34560 0 30.9mb 15.5mb
green open .monitoring-es-6-2018.12.09     AUSV3tD3QbSZ2KB48e-Ehg 1 1 34560 0 30.3mb 15.1mb
green open .monitoring-kibana-6-2018.12.07 V5VrjgA2SyONQORaVS_7Ow 1 1 34554 0 17.8mb  8.9mb
green open .monitoring-es-6-2018.12.03     bamPr41PTbqIJp8f6t5Lkg 1 1 34559 0 31.3mb 15.6mb
green open .monitoring-es-6-2018.12.07     _Rf-pl7gTXmRXWcqRSZTVw 1 1 34559 0 30.9mb 15.4mb
green open .monitoring-es-6-2018.12.10     doKfpZ-8Q8SGCuekLMuwcA 1 1 13412 0 14.9mb  7.4mb
green open .monitoring-kibana-6-2018.12.10 2KKxJBSzR--GZAjOT3-AaQ 1 1 13567 0  7.6mb  3.7mb
green open .monitoring-kibana-6-2018.12.08 tyRc08XERxe30bKuJhNWIA 1 1 34552 0 17.9mb    9mb
green open .monitoring-kibana-6-2018.12.04 gUVhR_EMTiivVzRoBltXsg 1 1 34552 0 17.6mb  8.8mb

(Chris Earle) #18

This may be something else, namely representative of this issue: https://github.com/elastic/kibana/issues/21013 .

The easiest way to confirm this is:

GET /.monitoring-es-6-*/_search
{
  "size": 0,
  "query": {
    "range": {
      "timestamp": {
        "gte": "now-1h"
      }
    }
  },
  "aggs": {
    "group_by_cluster": {
      "terms": {
        "field": "cluster_uuid"
      },
      "group_by_type": {
        "terms": {
          "field": "type"
        }
      }
    }
  }
}

I'm guessing that you are going to observe node_stats returned, but not cluster_stats. You can read about the causes of missing different types here based on the collector's type. The main page, and thus this screen, is driven by the requirement that we receive the cluster_stats document, which is the formative document for the cluster.

You've clearly enabled data and your indices prove that at least some of the data is making it through. The lack of deletes in the .monitoring-es-6-* also suggests that there are no shards documents, which is another thing that comes from the elected master and implies problems there (e.g., maybe you've configured it at the elasticsearch.yml level to communicate somewhere else).

Hope that helps,
Chris