Filebeat creates a "Standalone Cluster" in Kibana Monitoring

[Christophe_Journel (Chris)]

Hello,

I have a whole ELK stack which is perfectly functionnal on production.
However, i'm trying (with 7.2.0) to get metrics from filebeat.

The problem is: It creates a new "cluster" in Kibana Monitoring page : "Standalone Cluster"

I tried to delete /var/lib/filebeat/*, but i can make the filebeat to join my real production cluster.

Here is the configuration

monitoring:
enabled: true
elasticsearch:
hosts: ["host1:9200","host2:9200"]
username: remote_monitoring_user
password: xxxxxxxxxxxxxxxx

Thanks,

[DennisLoos]

I have the same behaviour with a similar configuration after upgrading the cluster and Beats to 7.2.0

[shaunak (Shaunak Kashyap)]

Hi @Christophe_Journel and @DennisLoos,

In 7.2, we allowed Filebeat to ship its monitoring data directly to a Monitoring Elasticsearch cluster (which might be separate from your Production Elasticsearch cluster, where you send the log data being harvested by Filebeat). The new monitoring.* settings introduced in 7.2 let you set the Monitoring cluster for Filebeat.

Now, if your output is elasticsearch, then we know which Elasticsearch cluster your Filebeat is sending its logs data to. In that case the Monitoring UI can correctly associate the Filebeat instance with that Elasticsearch cluster.

However, if the output is something other than elasticsearch, we cannot know if the logs data is ending up in an Elasticsearch cluster. So the Monitoring UI shows the Filebeat instance in a "Standalone Cluster".

[DennisLoos]

Hi Shaunak,

In my case, that's exactly what is happening. My output is sent to logstash for grokking and the monitoring is send to the Elasticsearch cluster. I have 3 Beats that have output to Elasticsearch and they appear in the cluster monitoring info.

Is there a way to connect them to the right cluster in the filebeat configuration or otherwise ?
That would also eliminate the cluster choice when opening monitoring.

[shaunak (Shaunak Kashyap)]

As a temporary workaround, you can go back to using the deprecated xpack.monitoring.* settings (instead of the new monitoring.* ones). However, please note that the xpack.monitoring.* settings will eventually go away in a future major version.

[ThomasMarcelis]

Hi Shaunak,

I also have the exact same scenario here. Are there plans for a permanent fix or workaround in the future?

[shaunak (Shaunak Kashyap)]

Hi @ThomasMarcelis,

Yes, we're discussing various options. Once we have a clear direction, I'll post a link to a GitHub issue/PR here so you can follow along.

Shaunak

[sergiodcm00]

I have the same problem.

[shaunak (Shaunak Kashyap)]

I just put up a PR for a proposed solution here: https://github.com/elastic/beats/pull/13182.