# Deduplicate data

**URL:** <https://discuss.elastic.co/t/deduplicate-data/307372>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 16, 2022, 9:31am UTC](https://discuss.elastic.co/t/deduplicate-data/307372 "2022-06-16T09:31:14Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Worlock](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Worlock](https://discuss.elastic.co/u/Worlock)\
**Post date:** [June 16, 2022, 9:31am UTC](https://discuss.elastic.co/t/deduplicate-data/307372/1 "2022-06-16T09:31:14Z")

</div>

So I have data that is getting exported to a JSON file, that's being uploaded to Elastic using FIlebeat.  
One of the fields can change, and when that happens, I would like the record being updated in Elastic instead of uploading a new entry, which is the case now.

In [this topic](https://discuss.elastic.co/t/filebeat-and-updating-documents/167355) I found some information on this, it's called deduplication, and logically you can achieve that by giving the two records the same ID.

I added this this to my filebeat.yml:

```auto
filebeat.inputs:
- type: filestream
  json.document_id: "AlertId"

```

AlertId is a unique ID that's within the data, so I would like to use that as the document ID.  
But this does not seem to work. Elastic still generates it's own ID. Can anyone explain what I have to do, to get this working?

---

<div class="post-metadata">

**Author:** ![TiagoQueiroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tiagoqueiroz/32/107061_2.png) [@TiagoQueiroz](https://discuss.elastic.co/u/TiagoQueiroz)\
**Post date:** [June 17, 2022, 10:52am UTC](https://discuss.elastic.co/t/deduplicate-data/307372/2 "2022-06-17T10:52:49Z")

</div>

On filestream you have to use the `ndjson` parser to set the ID:

```nohighlight
filebeat.inputs:
  - type: filestream
    id: "my-unique-filestream-id"
    paths: /tmp/flog/*.log
    parsers:
      - ndjson:
          document_id: "my-id-field"

```

or the `decode_json_fields` processor:

```nohighlight
filebeat.inputs:
  - type: filestream
    id: "my-other-unique-filestream-id"
    paths: /tmp/flog/*.log
    processors:
      - decode_json_fields:
          document_id: "my-id-field"
          fields: ["message"]
          max_depth: 1
          target: ""

```

---

<div class="post-metadata">

**Author:** ![Worlock](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Worlock](https://discuss.elastic.co/u/Worlock)\
**Post date:** [June 17, 2022, 11:38am UTC](https://discuss.elastic.co/t/deduplicate-data/307372/3 "2022-06-17T11:38:02Z")

</div>

Thank you! Works perfectly now.

---

<div class="post-metadata">

**Author:** ![Worlock](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Worlock](https://discuss.elastic.co/u/Worlock)\
**Post date:** [June 17, 2022, 12:08pm UTC](https://discuss.elastic.co/t/deduplicate-data/307372/4 "2022-06-17T12:08:37Z")

</div>

I spook a little too soon. The document id is now correct. But what I expected to happen, does not happen. When one field changes, this does not get changed in Elastic now, but there is also no new entry. It looks like filebeat/elastic is ignoring it now somehow.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 15, 2022, 2:08pm UTC](https://discuss.elastic.co/t/deduplicate-data/307372/5 "2022-07-15T14:08:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.