Logstash monitoring not working properly

Elastic Stack Logstash
1

Hello, I am not sure which category I should use for this thread but since it is monitoring related I will use X-Pack.

So I have 4 Logstash 5.6.1 instances running and 5 Elasticsearch 5.6.1 nodes all running X-Pack 5.6.1...

Now in Kibana monitoring I can only see 2 Logstash instances although in reality all 4 are running perfectly fine without any errors. But then this gets really weird. The instance Nr. 1 is always shown in Kibana monitoring but the rest are just iterating through each other on each page reload. So it means that all 4 instances are being monitored but somehow only 2 can be displayed in Kibana at the same time so it just cycles through the instances for the second spot. Any suggestions? :smiley:

What I tied so far:

  1. Deleting the Logstash monitoring index.
  2. xpack.monitoring.elasticsearch.url option set from array to a single Elasticsearch data node.
2

How did you install Logstash?

3

Offline debian package. At first I was using Logstash 5.2.2 without X-Pack but now I upgraded to Logstash 5.6.1 and installed X-Pack. Note that everything was done offline because it is an isolated cluster.

4

Also I checked all the documents in the Logstash monitoring index and everything looks OK. All 4 instances appear in the index as they should so to me it looks more and more like Kibana issue.

5

Ah I think I see the problem. I checked the monitoring index again and its strange because the instance Nr. 1 has unique "uuid": "12dab77e-a7f6-4560-93d4-be249d47b08a" and the other three have the same "uuid": "6bacfeba-4955-43fc-863d-75c37a94ec7a". Could this be it? And if so why and how did this happen...

6

That was what I was looking for. I think UUID is generated on installation. Did you install Logstash on each machine or clone the directory/machine image for some of them?

7

Now that is a good question. Sadly I was not the one who did the initial installation. All I know is that 2 instances were setup at first and 2 were added later. So I am pretty sure they might have cloned the instance Nr.2 when adding 2 new ones...

8

That would explain it...

9

Now I have a feeling that there is no way to change the uuid without performing a full reinstall? :slight_smile:

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.