Disable cluster setting: xpack.monitoring.enabled

Elastic Stack Elasticsearch
1

Setup: Elasticsearch 6.3.0 with 19 nodes:

12 data nodes
4 client nodes
3 master nodes

We have around:

Cluster size: ~820GB (~175GB of them are .monitoring indices)
1,030,000,000 documents in the cluster (900,000,000 are in .monitoring documents)
7,100 indices (7 of them are .monitoring indices)
24,300 shards (14 of them are .monitoring shards)
xpack.monitoring.enabled os set to true (default).
Also my xpack.monitoring.collection.enabled is set to true (historically, I used Kibana to monitor but I'm not using it anymore).
I am not using those indices and I want to set them both to false.

Both settings are static and raise the following questions:

  • Every node contains elasticsearch.yml file. Which one is in charge of the cluster global settings? - will change the master node settings is enough?
  • Is it safe to do it "one node at a time"? this is a production cluster and I can't afford downtime.
  • Which one of those flags is responsible for this high document rate?

Thanks,
Itay

2

That setting is a so called dynamic setting, which means it can be changed at runtime using the Cluster Update Settings API.

This way you can prevent changing the setting on every node and restarting (you still may want to change the configuration file in your VCS after running that change, just for clarity, however the persistent cluster state will always take precedence, even in a full cluster restart).

3

Hi @spinscale,

Thanks for your reply.
Not sure how I missed that xpack.monitoring.collection.enabled is dynamic!
Are you sure about xpack.monitoring.enabled?
https://www.elastic.co/guide/en/elasticsearch/reference/current/monitoring-settings.html#general-monitoring-settings
I don't see that it's dynamic settings and in cerebro it's mentioned as static settings:
image

4

Side note - That's wwwwwwaaaaaaayyyyyy too many. You really need to look at reducing that count.

2 Likes
5

Noted, will do. Thanks @warkolm

6

That setting is not dynamic - but does it matter, when the collection can be disabled? I mean, you can just wait for the next node restart then, I would assume and disable the setting in that case?

7

I already disabled xpack.monitoring.collection.enabled thanks to your advice (I thought it static).
Now I'm trying to understand how to disable xpack.monitoring.enabled.
If the current value (in elasticsearch.yml) of xpack.monitoring.enabled is true and I'll change it in one single arbitrary instance (I will edit the elasticsearch.yml and set this value to false). Will it apply to the entire cluster?

8

no, it will apply to each node and not be applied via cluster state. basically setting that setting to false will not start the internal collector services (not even instantiate the classes) or register any monitoring specific REST actions (things that are all done before any network connectivity is happening).

hope that helps.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.