ELK Monitoring Cluster - issue with indexes reaching

[dominbdg (Dominik)]

Hello,
I have following issue.
I created ELK Onenode cluster with self-monitoring enabled (as ELK Monitoring Cluster)
I joined another ELK Cluster environment to Monitoring Cluster.
From another ELK Cluster I configured metricbeats to connect to elasticsearch from Monitoring Cluster and elasticsearch-xpack module to access locally elasticcsearch for monitoring.

My environment is configured that way that ELK Cluster is sending data to elasticsearch in monitoring Cluster using port 9200. This is only one port enabled between servers.

My issue is - when from Monitoring Cluster I'm going to monitored ELK Cluster and want to see indexes, after couple of minutes I'm receiving error 504.

The question is - should I configure something more there ?
I was looking about to configure beats-xpack module - will I need it to successfull communication ?

[stephenb (Stephen Brown)]

@dominbdg

monitoring more than 1 cluster with a single monitoring cluster is a commercially licensed feature (Multi-Stack Monitoring), perhaps that is the issue if you are using a Basic / Free license, See here

What license are you running? You can also check the elasticsearch or metricbeat logs as well.

[dominbdg (Dominik)]

no, this is not this issue because right now I have one cluster connected to monitoring - both of them have platinum license.

my issue is that - when from cluster with monitoring I'm going to elasticsearch of monitored cluster, click indices - it is searching and .. error 504 . Maybe this is related with some timeout ?

[stephenb (Stephen Brown)]

Apologies I am still a little unclear ...

What version is everything.

Perhaps a screenshot AND you should definitely look at the logs of the elasticsearch cluster that is the monitoring cluster.

@dominbdg

cluster with self-monitoring enabled (as ELK Monitoring Cluster)
I joined another ELK Cluster environment to Monitoring Cluster.

Also if I am reading this right 1 cluster is self-monitoring (deprecated) and one cluster is using metricbeat easlticsearch-xpack module (go forward method).... not sure that is the best setup.... that could be a problem as well and I believe that monitoring indices are different.

Not saying that IS the issue... but not sure if you can mix them.

Self Monitoring Indices...

Metricbeat Monitoring

Datastream + backing indices

[dominbdg (Dominik)]

ok,
You said that this is not correct setup.
May I ask how should I implement monitoring ?

Let me explain my implementation:
On one server I installed called as ELK Monitoring Cluster - elasticsearch and kibana.
I choosed under monitoring tab in kibana - enable self monitoring

On the another server which I would like to have be monitored:
in elasticsearch I have installed metricbeat.
in /etc/metricbeat/metricbeat.yml I included elasticsearch configuration (host,username,password) points to ELK Monitoring.
enabled elasticsearch-xpack.yml in modules,
in /etc/metricbeat/modules/elasticsearch.yml included configuration of local elasticsearch.

In monitoring cluster I see monitoring environment, all looks like fine,
but when I click on monitored environment -> elasticsearch -> indices
it is looking for indices and after couple of minutes cannot find anythhing and throws 504

I have a question also - if my implementation of monitoring is correct or I should change it.

[stephenb (Stephen Brown)]

What Version?

I did not say your configuration is not correct. I just said it could be an issue... And that is not how we would typically suggest monitoring more than 1 cluster.

@dominbdg

May I ask how should I implement monitoring ?

Perhaps take a look at the documentation here and here

You could monitor both with a single metricbeat module / single metricbeat instance... you can put more than 1 cluster in the elasticsearch-xpack.yml

I just did this and it works as expected.

example

- module: elasticsearch
  xpack.enabled: true
  scope: cluster
  period: 10s
  hosts: ["http://localhost:9200"]
  metricsets:
    - node
    - node_stats
    - index
    - index_recovery
    - index_summary
    - ingest_pipeline
    - shard
    - ml_job

- module: elasticsearch
  xpack.enabled: true
  scope: cluster
  period: 10s
  hosts: ["http://otherhost:9200"]
  metricsets:
    - node
    - node_stats
    - index
    - index_recovery
    - index_summary
    - ingest_pipeline
    - shard
    - ml_job

However now that you have enabled self monitoring ...
xpack.monitoring.collection : true

I think you are going to need to disable that false you can take a look at the settings here

Why you are getting a 504 I have not clue... you would need to look at the logs.

[dominbdg (Dominik)]

maybe there is my issue.

My code of elasticsearch module don't cover things You showed me :

- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["http://localhost:9200"]
  username: "elastic"
  password: "elastic-password"

I will try to implement elasticsearch-xpack like You showed me

[stephenb (Stephen Brown)]

Again....what version?... There has been changes

Make sure you look at the correct documentation and the correct module

Ohh and you might need to clean up the old monitoring indices

[dominbdg (Dominik)]

I'm using latest version,
both environments are running version 8.8.1