Stack monitoring when using Pipeline to Pipeline communication

Hi all!

Stack version: 7.6.0

Using the new remote monitoring (monitoring the elastic stack using metricbeat) I bumped into something that I can't find a solution for.

In the Stack Monitoring view in Kibana I now, after configuring Pipeline to Pipeline communication, always get two clusters: My actual cluster and the "Standalone Cluster".

In the Standalone Cluster I can see my single logstash node (that is using Pipeline to Pipeline communication) together with one (1) pipeline; the pipeline that is using the pipeline output. Note that the same logstash node is also included in my actual cluster, but here with only the other pipeline, the one using the elasticsearch output plugin.

My Pipelines:

input-beats.conf - Standalone Cluster

    input {
      beats {
        port => 5044
      }
    }

    output {
      pipeline {
        send_to => ["output-elasticsearch"]
      }
    }

output-elasticsearch.conf - Correct cluster

    input {
      pipeline {
        address => "output-elasticsearch"
      }
    }

    output {
      elasticsearch{
        # my output config
      }
    }

So, the data is arriving and all is well. But why isn't the first pipeline recognized as part of my actual cluster?
I'm guessing that the elasticsearch output plugin finds the cluster.uuid automatically and uses that to identify with the cluster, but isn't there any way for my input pipeline to do this as well? Or at least a way to tell it which cluster it belongs to in the configuration?

Can't seem to find an answer to this after some heavy reading on documentation and google.

Hi @niklaskurvinen,

Thanks for raising this issue!

I managed to successfully reproduce this locally and there is a work around coming: https://github.com/elastic/logstash/pull/11106

With this PR, you'll be able to add a new setting monitoring.cluster_uuid which will ensure all monitoring data is stored under that cluster which will mean you won't see the Standalone Cluster anymore.

Lemme know if that helps

1 Like

Thanks @chrisronline!

Figured that there would be a PR somewhere, that's perfect!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.