Google Workspace module using wrong field to avoid duplicates

Elastic Stack Beats
1

Hi,

I have recently noticed that Google Workspace module uses @metadata.id instead of @metadata._id. It leads to duplicate appearance in logs.

Changing this:

var addID = new processor.Fingerprint({
    fields: [
        "json.id.time",
        "json.id.uniqueQualifier",
        "json.id.applicationName",
        "json.id.customerId",
    ],
    target_field: "@metadata.id",
    ignore_missing: true,
    fail_on_error: false,
});

to this:

var addID = new processor.Fingerprint({
    fields: [
        "json.id.time",
        "json.id.uniqueQualifier",
        "json.id.applicationName",
        "json.id.customerId",
    ],
    target_field: "@metadata._id",
    ignore_missing: true,
    fail_on_error: false,
});

should fix the problem.

2

Testing this now on Filebeat 7.17.4. I noticed a significant increase in messages on June 2 through today. Ton of duplicate messages.

3

So far it appears better. Will test for a few days.

4

Here is a pull request that will help to fix this issue in future releases.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.