# How to prevent duplicates in filebeat

**URL:** <https://discuss.elastic.co/t/how-to-prevent-duplicates-in-filebeat/296391>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 6, 2022, 11:13am UTC](https://discuss.elastic.co/t/how-to-prevent-duplicates-in-filebeat/296391 "2022-02-06T11:13:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dror\_Tirosh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dror_tirosh/32/98379_2.png) [@Dror\_Tirosh](https://discuss.elastic.co/u/Dror_Tirosh)\
**Post date:** [February 6, 2022, 11:13am UTC](https://discuss.elastic.co/t/how-to-prevent-duplicates-in-filebeat/296391/1 "2022-02-06T11:13:35Z")

</div>

My input data (which is json-formatted) as a unique field.  
According to the [documentation](https://www.elastic.co/guide/en/beats/filebeat/8.0/filebeat-deduplication.html), this should use it as a unique identifier, and thus pervent duplicates:

```auto
processor:
- fingerprint:
    fields: ["tx"]
    target_field: "@metadata._id"

```

However, it doesn't help.  
I did verify (in debug mode) that I have a unique value set in the `@metadata: { _id: "xxx" }`  
(it is not a copy of my field, but probably a hash on it: verified that sending multiple times the same record generates the same ID)

when processing copies of a file, it generates more records, instead of de-duping them.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 7, 2022, 12:52am UTC](https://discuss.elastic.co/t/how-to-prevent-duplicates-in-filebeat/296391/2 "2022-02-07T00:52:49Z")

</div>

What method tool are you using to ingest / index your documents?

And are you saying your are seeing multiple documents in the same index with the same `_id`?

If so please show us that.

From the docs looks like you're doing the right thing.

EDIT I just ran a file through Filebeat directly to Elasticsearch with with duplicates on my fingerprint fields and this config and it worked as expected no duplicates ... so perhaps tell us more about the rest of you ingestion

Just for example I just put these lines in a "log" file

```auto
Apple
Dell
Google
Apple
Amazon
Amazon
Amazon
Google
Dell
Google
Google
Google

```

These fields get interpreted as the `message` field

I used this

```auto
  - fingerprint:
      fields: ["message"]
      target_field: "@metadata._id"

```

and this is my result, notice only 4 documents and they have the fingerprint as the `_id`

```auto
GET filebeat-7.16.3-2022.02.07-000001/_search

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "filebeat-7.16.3-2022.02.07-000001",
        "_type" : "_doc",
        "_id" : "f0aa2b6382e5f554aeb83a966a6148c988a9a0553d04d4bcd06747b2bac21d74",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2022-02-07T01:24:42.121Z",
          "ecs" : {
            "version" : "1.12.0"
          },
          "container" : {
            "id" : "7.16.3"
          },
          "log" : {
            "offset" : 6,
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.16.3/filebeat-7.16.3-darwin-x86_64/test.log"
            }
          },
          "message" : "Apple",
          "input" : {
            "type" : "filestream"
          },
          "agent" : {
            "id" : "04373b11-e7ab-4632-8e54-e30a2d3f8227",
            "name" : "hyperion",
            "type" : "filebeat",
            "version" : "7.16.3",
            "hostname" : "hyperion",
            "ephemeral_id" : "fa7650a3-4e5d-44b6-b408-c57402a6c93f"
          }
        }
      },
      {
        "_index" : "filebeat-7.16.3-2022.02.07-000001",
        "_type" : "_doc",
        "_id" : "52fadd3719f7ae3eb341061bfedc2653118c464cd75aff2bf10d6e42d9b32f92",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2022-02-07T01:24:42.121Z",
          "ecs" : {
            "version" : "1.12.0"
          },
          "agent" : {
            "ephemeral_id" : "fa7650a3-4e5d-44b6-b408-c57402a6c93f",
            "id" : "04373b11-e7ab-4632-8e54-e30a2d3f8227",
            "name" : "hyperion",
            "type" : "filebeat",
            "version" : "7.16.3",
            "hostname" : "hyperion"
          },
          "container" : {
            "id" : "7.16.3"
          },
          "log" : {
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.16.3/filebeat-7.16.3-darwin-x86_64/test.log"
            },
            "offset" : 11
          },
          "message" : "Dell",
          "input" : {
            "type" : "filestream"
          }
        }
      },
      {
        "_index" : "filebeat-7.16.3-2022.02.07-000001",
        "_type" : "_doc",
        "_id" : "29aea56777a90acd076e75cf91aef1fe4c5d8bd2b5786194fc7b5ef7dd9a76c7",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2022-02-07T01:24:42.121Z",
          "input" : {
            "type" : "filestream"
          },
          "ecs" : {
            "version" : "1.12.0"
          },
          "agent" : {
            "version" : "7.16.3",
            "hostname" : "hyperion",
            "ephemeral_id" : "fa7650a3-4e5d-44b6-b408-c57402a6c93f",
            "id" : "04373b11-e7ab-4632-8e54-e30a2d3f8227",
            "name" : "hyperion",
            "type" : "filebeat"
          },
          "container" : {
            "id" : "7.16.3"
          },
          "log" : {
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.16.3/filebeat-7.16.3-darwin-x86_64/test.log"
            },
            "offset" : 18
          },
          "message" : "Google"
        }
      },
      {
        "_index" : "filebeat-7.16.3-2022.02.07-000001",
        "_type" : "_doc",
        "_id" : "c5a60e4fed8743f73ec171491c06f1e991b20880e57ced4f0d8de0cc2106f692",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2022-02-07T01:24:42.121Z",
          "container" : {
            "id" : "7.16.3"
          },
          "log" : {
            "file" : {
              "path" : "/Users/sbrown/workspace/elastic-install/7.16.3/filebeat-7.16.3-darwin-x86_64/test.log"
            },
            "offset" : 31
          },
          "message" : "Amazon",
          "input" : {
            "type" : "filestream"
          },
          "ecs" : {
            "version" : "1.12.0"
          },
          "agent" : {
            "ephemeral_id" : "fa7650a3-4e5d-44b6-b408-c57402a6c93f",
            "id" : "04373b11-e7ab-4632-8e54-e30a2d3f8227",
            "name" : "hyperion",
            "type" : "filebeat",
            "version" : "7.16.3",
            "hostname" : "hyperion"
          }
        }
      }
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![Dror\_Tirosh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dror_tirosh/32/98379_2.png) [@Dror\_Tirosh](https://discuss.elastic.co/u/Dror_Tirosh)\
**Post date:** [February 7, 2022, 10:46am UTC](https://discuss.elastic.co/t/how-to-prevent-duplicates-in-filebeat/296391/3 "2022-02-07T10:46:42Z")

</div>

not sure, but now it works, and entries are de-duped correctly, with the same configuration.  
thanks anyway.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 7, 2022, 12:47pm UTC](https://discuss.elastic.co/t/how-to-prevent-duplicates-in-filebeat/296391/4 "2022-03-07T12:47:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.