Logstash Monitoring API -

Elastic Stack Logstash
1

Hi Guys!

i have 3 logstash servers installed for my data pipeline.

i have enabled the xpack.monitoring for each one of the servers, when i use Kibana and go to "Monitoring" tab, i see only one instance although i have 3.

each time i ran refresh i see another logstash instance, logstash version:6.8.

i have tried already to check if there is any content under /usr/share/logstash/data directory
but it was empty of context.

Any advice?

Thanks,
Gidi

2

Hi @Gidi_Kalef,

each time i ran refresh i see another logstash instance

So for example, if you have logstash instances A, B, C, you're saying that every time you refresh, you see either A or B or C? But not all three?

3

yes! exactly

4

Do you recall how you setup each logstash instance? Did you fresh install them all? Or did you copy/paste from one to the other?

5

I have cloned the machines

6

You probably have duplicated uuids across all logstash instances. We have this enhancement request to improve this in the UI, but for now, you'll need to manually remove the logstash/data/uuid file (logstash should recreate this on restart) for each logstash instance and restart each one.

7

can you please refer me to the relative path?
I have already checked /usr/share/logstash/data and there is no such a file.

thanks.!

8

Ah, it's possible that file doesn't exist in your version (6.8). Are you able to safely delete the data/ directory?

9

yes I'm able, so I should delete /usr/share/logstash/data dir?

10

Yes, try deleting them and seeing if that fixes the issue

11

Unfortunately, it doesnt helps :frowning:
Is there another suggestions?

12

Hmm.

Try running this query against the monitoring cluster and report back the results:

POST .monitoring-logstash-*/_search
{
  "size": 10,
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "timestamp": {
              "gte": "now-15m"
            }
          }
        },
        {
          "term": {
            "type": {
              "value": "logstash_stats"
            }
          }
        }
      ]
    }
  },
  "collapse": {
    "field": "logstash_stats.logstash.uuid"
  }
}
13

I had a similar problem a while back and got good advice here along the same lines.

I was generating AMIs and starting them on AWS EC2, so I implemented this "user data" script that runs when each Logstash instance starts.

#!/bin/bash

# In order for Elastic Cloud to be able to monitor more than one Logstash host, the UUIDs must be unique
# across all Logstash hosts which are feeding a given Elasticsearch cluster.
# https://discuss.elastic.co/t/x-pack-monitoring-shows-one-of-all-of-the-configured-logstash-nodes/142760
# The file /var/lib/logstash/uuid must be created on instance startup, before logstash is started.
# Deleting /var/lib/logstash/uuid at the time the AMI is generated is ineffective, because it is not
# generated until Logstash starts for the first time.

UUID=`uuidgen | tr -d "\n\r" `
echo "generated Logstash UUID ${UUID}"

echo -n ${UUID} > /var/lib/logstash/uuid
chown logstash:logstash /var/lib/logstash/uuid
echo "wrote Logstash UUID to file /var/lib/logstash/uuid"
14

@chrisronline

Hi ,
i have founded the soultion thanks to @tomj ,
i have looked in the wrong path, i had should looked for in /var/lib/logstash and i have searched under /usr/share/logstash/data instead

Thanks everyone!

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.