Duplicate beat.uuids on different hosts

[rozling]

Hi,

I'll preface this by saying I have a ticket open with Elastic Support about this, but since it's a really weird one IMO, I thought'd I'd post here in case anyone else has seen it.

Basically we're sending monitoring data for about 70 beats to our monitoring cluster, and only ~12 are showing up.

After some investigation (by checking the .monitoring-beats-* index) we realised that many beats on different hosts have identical UUIDs

The monitoring UI sees each UUID as one beat, although when the page refreshes the last beat.name seen for that UUID is displayed.

Some facts:

• Metricbeat / Filebeat v 7.5.0
• Dedicated Monitoring Cluster (Elastic Cloud based) v7.5.1
• Amazon Linux v2
• Systemd

We're deploying beats via an Ansible playbook and to ensure unique beat.names we're doing a couple of things:

• Generating unique hostnames using a bash script. Hostnames looks something like dub-myinstance-50-102
• Storing this variable in /etc/environment as $HOSTNAME_SHORT
• Using a systemd script to run the service which sets EnvironmentFile=/etc/environment. This also sets LimitNOFILE=none
• We also use autodiscover to gather metrics for docker containers

I have no idea how two separate instances of e.g. Metricbeat, on two separate hosts, can generate the same UUID

This happens even if we download the package and untar the contents over the existing installation in /usr/share/metricbeat!

I'd be interested to know how the UUID is generated, i.e. whether it's some underlying OS thing which might be customised for Amazon Linux V2

Any help / thoughts appreciated!

[ropc (Romain Chanu)]

Hi @rozling,

1. Could you share with me the support case number so I can check the details?

2. This is my understanding of how the beat ID is currently generated / set:

• When you run a beat instance for the first time, a beat ID will be generated by calling uuid.NewV4(). This function is defined in the uuid package.
• The beat ID will then be stored in the meta.json file (stored in {path.home}/data folder) so that the UUID can be reloaded the next time you run the beat instance.

Regarding the UUID generation itself, I might need to check with the team if you wish to have further details.

For now, I am just wondering if the problem does not rely in the way you deploy beats. Could it be possible that the same meta.json file is being copied over across multiple hosts? If somehow, the beats instances use the same meta.json file (which contains the UUID), that would explain this situation. It is just a rough guess, but it could worth checking this.

[rozling]

Hi ropc,

Thanks for your reply, this was really helpful!

I'm not sure how meta.json could have been the same - I don't think we are deploying anything there but am looking into it.

However I tried deleting the meta.json and straightaway it generated unique UUIDs for each host. Both UUIDs are now showing up in the .monitoring-beats index, but not in the UI which is interesting. I have about 15 minutes of data so far, but even with the monitoring UI set to 5 minutes I don't see anything.

I'll send you the case number in a PM.

Thanks again!

[ropc (Romain Chanu)]

You're welcome @rozling Do let us know if you find anything related to the deployment and the meta.json files. I am glad that we managed to have some progress in your issue. Please do follow-up with our support engineer on the other issue with the monitoring UI. Thank you!