Stack Monitoring not working properly

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I am working currently on getting the stack monitoring up and running.
According to what I have read so far, all I need to do for that is set the xpack.monitoring.collection.enabled to true. So like that:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elastic
spec:
  version: {{ $.Chart.AppVersion }}
  secureSettings:
  - secretName: elastic-es-gcs-credentials
  nodeSets:
  - name: master
    count: 3
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.ml: false
      node.store.allow_mmap: false
      xpack.monitoring.collection.enabled: true
....

Same is enabled for the data nodes and for kibana.
On the first run it looked pretty good. However, after a few days I go again the default via:

image
image822×474 23.8 KB

I check that the parameters are still in place and the indicies exist:

image
image1194×545 63.5 KB

As we can see everything is there. However, Kibana says no monitoring data are there. If I go an delete the monitoring indices of for example today manually, it works again... until it stops in one or two days again...

Any idea what could be going wrong here?

2

Go thru the "self monitoring" setup and see if that fixes it. xpack monitoring is "self monitoring", I think that is being replaced with metricbeat modules for each component. I'm not sure it's "better" yet :slight_smile:

3

I just had the same problem. My guess is that you have created an index template under the * pattern that changes the number of shards for all new indices. At least in my case, when either of the monitoring indices had more than 1 shard, it would not appear at all in the monitoring page. To confirm that, I deleted the pattern and the index(make sure you have backups or don't need the data), and as soon as the new index was created with 1 shard/1 replica, the monitoring page works again.
You can also confirm if this is the issue by increasing the time range until the last time that you remember the monitoring working.
To solve it, I created a different index template for the monitoring indices with no configuration, and reduced the priority of the one with the * pattern.

1 Like