# Duplicated date in my elastic

**URL:** <https://discuss.elastic.co/t/duplicated-date-in-my-elastic/315506>\
**Category:** Logstash\
**Created:** [September 30, 2022, 12:00am UTC](https://discuss.elastic.co/t/duplicated-date-in-my-elastic/315506 "2022-09-30T00:00:37Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\
**Post date:** [September 30, 2022, 12:00am UTC](https://discuss.elastic.co/t/duplicated-date-in-my-elastic/315506/1 "2022-09-30T00:00:37Z")

</div>

Hello Team,  
Greeting,

I have a situation when the data is being pushing into my elastic twice, a duplicate data.  
now both of them have different id and other than that everything is same. How do I get over this.

My filebeat

```auto
filebeat.inputs:
- type: log
  fields_under_root: true
  exclude_lines: ["^$"]
  tail_files: true
  paths:
  - '/var/www/*.log'

processors:
  - add_id: ~
  - add_tags:
      tags: [XXX]
      target: "application"

```

and in my Logstash output I have

```auto
else if [@metadata][_id] and "XXX" in [application] {
        elasticsearch {
        hosts => ["https://es:9200"]
        index => "write-alias"
        user => "log-user"
        }
    }

```

Can you guys please help me on this?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 30, 2022, 12:26am UTC](https://discuss.elastic.co/t/duplicated-date-in-my-elastic/315506/2 "2022-09-30T00:26:08Z")

</div>

You need to share your entire logstash configuration, it is not possible to know what could be the issue with only this part.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 30, 2022, 3:32am UTC](https://discuss.elastic.co/t/duplicated-date-in-my-elastic/315506/3 "2022-09-30T03:32:24Z")

</div>

> [@adityak248](#):
>
> ```auto
> processors:
> - add_id: ~
> 
> ```

The add\_id processor in filebeat adds a unique value for [@metadata][\_id]. You want duplicate events to have the same id, not a unique one. I would suggest a [fingerprint](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html) filter, and configuring it to use a hash (not MAC) of whatever set of fields you think make an event unique.

---

<div class="post-metadata">

**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\
**Post date:** [September 30, 2022, 5:16pm UTC](https://discuss.elastic.co/t/duplicated-date-in-my-elastic/315506/4 "2022-09-30T17:16:34Z")

</div>

Hello Badger, thanks for the response.  
Here is what I have right now. when log is generated there is specific req id that get created and that is associated to couple of logs, something like this,

```auto
abc123xyz [time] Started......
abc123xyz [time] Processing......
abc123xyz [time] some_application_log......
abc123xyz [time] Completed......

```

Time is same, reqid is same and each line has different fields no unique one.  
How do you think I should proceed in this case.

Thanks

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 30, 2022, 5:34pm UTC](https://discuss.elastic.co/t/duplicated-date-in-my-elastic/315506/5 "2022-09-30T17:34:45Z")

</div>

You can fingerprint the entire [message] field, which is what the filter does by default.

```
fingerprint { target => "[@metadata][id]" method => SHA256 }

```

then use `document_id => "%{[@metadata][id]}"` in the elasticsearch output.

---

<div class="post-metadata">

**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\
**Post date:** [October 4, 2022, 10:56pm UTC](https://discuss.elastic.co/t/duplicated-date-in-my-elastic/315506/6 "2022-10-04T22:56:08Z")

</div>

Hello Badger, thanks for the response. Can you help me with this.

```auto
1d67c3b0 I, [2022-10-04T22:46:46.898239 #3722] INFO -- : Log-MESSAGE

```

After observing the logs the only field that is uniques I recognized is "2022-10-04T22:46:46.898239" with all 6 decimals.  
How do I extract the that field to associate as a unique id. Here is what the documentation says on applying processors in Filebeat.

```auto
processors:
  - fingerprint:
      fields: ["field1", "field2"]
      target_field: "@metadata._id"

```

I guess its not that easy to mention something like

```auto
processors:
  - fingerprint:
      fields: ["timestamp"]
      target_field: "@metadata._id"

```

We might end up using dissect or what not. can you please help me here.  
Thanks in advance.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 1, 2022, 10:56pm UTC](https://discuss.elastic.co/t/duplicated-date-in-my-elastic/315506/7 "2022-11-01T22:56:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.