Missing x-pack filebeat monitors

Elastic Stack Elasticsearch
1

Hello

In the beginning our ELK stack filebeat monitoring worked perfectly with X-Pack but recently I tried adding some new hosts to it and although the data is correctly parsed by Logstash the monitoring data that get's send directly to ElasticSearch does something weird.

Instead of creating a new filebeat instance in the Beats monitoring it overwrites an old one which causes us to not be able to monitor all the filebeats. When I then restart the previous filebeat the hostname of the filebeat instance get's changed back and the new one isn't monitored anymore.

First I tried to fix this by adding the 'name' tag to the filebeat configuration file but that doesn't really do anything. I know the issue is probably caused because these servers were cloned and at that time filebeat was running but I don't get why I can't fix it.

Is there anything I am missing, perhaps a configuration option?
Or do you require more data to situate the problem?

Thank you for your help

Jens

1 Like
2

After some further researching I found that the UUID of these filebeats are exactly the same so I suppose that's were the error lies but how can I fix this, in other words how does the uuid get created for a filebeat.

1 Like
3

After some further researching I found that the UUID of these filebeats are exactly the same so I suppose that's were the error lies but how can I fix this, in other words how does the uuid get created for a filebeat.

This is indeed the issue. The same issue can exist for Elasticsearch, Kibana, and Logstash for the same reason.

The Persistent UUID for every component in the stack is created in their path.data directory, which always defaults to ${HOME}/data (which tends to be ./data), on initial startup. For Elasticsearch, this is less of an issue because a cluster will actively block two nodes from joining with the same persistent UUID, but other components in the stack are not clustered and thus do not know about this collision.

The issue generally presents itself when people create Beats, Logstash, and Kibana configurations in development, but then copy/paste those directories instead of completely reinstalling them for future copies. This will copy the ./data directory and copy the persistent UUID as a result.

The solution is to simply delete the ./data directory and restart the Filebeat instance. You can be more surgical about it by deleting the UUID file inside of the data directory. In the future, the ideal solution is to use clean installs and explicitly set the path.data property for installs so that upgrades are marked as upgrades rather than replacements.

1 Like
4

Same problem here.
In CENTOS with filebeat 6.3.1 the uuid of the filebeat module is in /var/lib/filebeat/meta.json
Removing this file and restarting de service recreates the file with a new uuid and problem solved.
Thank you guys.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.