Cannot see the production cluster data on the monitoring cluster

Elastic Stack Elasticsearch
1

Hi there! I've 2 elk clusters version 7.6.1 "Basic license", 1st -production which have 3nodes (elastic, kibana, logstash and metricbeat agents on each node) and 2nd - monitoring cluster which have 1 node (elastic, kibana, metricbeat agent)

the question is: can i see all the nodes from my production cluster on my monitoring cluster which have own kibana instance with "Basic license" ?
thanks!

2

You can monitor your production cluster with another monitoring cluster, you need to use metricbeat and configure it to send data to your monitoring cluster.

But you cannot self-monitoring your monitoring cluster as the basic license does not allow monitoring multiple clusters.

3

ok! i got it, but what is the right way to achieve this ? I've re-read all the docs several times, but still cannot see the nodes in my kibana on monitoring cluster side

i follow the instructions and setup the metricbeat on prod nodes cluster with the following lines:

  hosts: ["elk-mc.local:9200"]  <--- url of my monitoring cluster
  username: "elastic"
  password: "my_password"
  protocol: "https"
  ssl.enabled: true
  ssl.verification_mode: none
  ssl.certificate_authorities: ["/etc/metricbeat/elastic-stack-ca-elk-mc.pem"]
  • elasticsearch-xpack.yml with the following lines:
- module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - enrich
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["https://elk1.local:9200"] <--- first node of my prod cluster
  username: "elastic"
  password: "my_password"
  ssl.verification_mode: none
  ssl.certificate_authorities: ["/etc/metricbeat/elastic-stack-ca.pem"]
  xpack.enabled: true

the index named metricbeat-7.6.1-2024.04.15-000001 was created by elastic on monitoring cluster, but there aren't any data
and "No monitoring data found" in Kibana after I've disabled the self-monitoring.
maybe I've missed something?

4

Did you check this part of the documentation?

The elasticsearch module can be used to collect metrics shown in our Stack Monitoring UI in Kibana. To enable this usage, set xpack.enabled: true and remove any metricsets from the module’s configuration.

Try to remove the metricsets and add a the scope as cluster.

I have it working with this setting:

- module: elasticsearch
  xpack.enabled: true
  period: 15s
  scope: cluster
  hosts: ["https://PROD-MASTER:9200"]
  username: "USERNAME"
  password: "PASSWORD"
  ssl.enabled: true
  ssl.verification_mode: "none"
  ssl.certificate_authorities: ["/etc/elasticsearch/certs/ca/ca.crt"]
5

If config:


- module: elasticsearch
  metricsets:
    - node
    - node_stats
  period: 10s
  hosts: ["https://elk1.local:9200"]
  username: "elastic"
  password: "my_password"
  ssl.verification_mode: none
  ssl.certificate_authorities: ["/etc/metricbeat/elastic-stack-ca.pem"]
  scope: cluster
  xpack.enabled: true

I see in the output:
Error creating runner from config: 1 error: The elasticsearch module with xpack.enabled: true must have metricsets: [ccr enrich cluster_stats index index_recovery index_summary ml_job node_stats shard]

and...

If config:

- module: elasticsearch
  metricsets:
    - node
    - node_stats
  period: 10s
  hosts: ["https://elk1.local:9200"]
  username: "elastic"
  password: "my_password"
  ssl.verification_mode: none
  ssl.certificate_authorities: ["/etc/metricbeat/elastic-stack-ca.pem"]
  scope: cluster
  #xpack.enabled: true

I see new docs in the Discover section in Kibana from production node, but still no data in Stack Monitoring

6

Yeah, it seems that in 7.6 is different, you need to specify the metricsets.

7.6 is pretty old and not supported anymore.

What is the version of your monitoring cluster?

7

both versions are 7.6.1

8

maybe is there one more module that should be enabled to send needed data to Monitoring stack UI ?

9

I don't think so, if the module that you want is already enabled, then you should have the data.

Just to confirm, you followed this documentation? Can you double check all the settings, including the cluster settings?

Not sure what is the issue, it may be some bug that was already fixed on later version. 7.6 is really old and not supported anymore.

12

I've achieved my aim via update through update my monitoring cluster to 7.10.2 and metricbeat also to 7.10.2 versions.
But now i came across with this problem: how can i disable the "monitoring-cluster" node and monitor only "myhome cluster" with my Basic license?

scr1
scr11891×777 47.1 KB

13

This is what I mentioned here, you have the self-monitoring for your monitoring cluster enabled.

You need to disable it, try to run this on your monitoring cluster:

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}

Also, set xpack.monitoring.enabled in your monitoring cluster elasticsearch.yml to false.

14

set xpack.monitoring.enabled in your monitoring cluster elasticsearch.yml to false - Done

and my _cluster/settings output:

{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "elasticsearch" : {
          "collection" : {
            "enabled" : "true"
          }
        },
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : { }
}

is this correct?, i still cannot use myhome cluster

15

The setting xpack.monitoring.elasticsearch.collection.enabled needs to be set to false on your production cluster as well, this is mentioned in the end of the documentation.

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.elasticsearch.collection.enabled": false
  }
}

It should work, but I'm not sure if you will need to delete the data you have in your monitoring cluster.

As mentioned this version is pretty old, it is been years since I used it and some things changed on newer versions to make this process simpler.

16

15 mins later it worked and in the Stack Monitoring Panel i've now only my prod cluster, and i see all my nodes, thank you so much!