# Duplicated events

**URL:** <https://discuss.elastic.co/t/duplicated-events/341389>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 22, 2023, 3:07pm UTC](https://discuss.elastic.co/t/duplicated-events/341389 "2023-08-22T15:07:41Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mohammed\_Amine\_El\_ha](https://avatars.discourse-cdn.com/v4/letter/m/bbe5ce/32.png) [@Mohammed\_Amine\_El\_ha](https://discuss.elastic.co/u/Mohammed_Amine_El_ha)\
**Post date:** [August 22, 2023, 3:07pm UTC](https://discuss.elastic.co/t/duplicated-events/341389/1 "2023-08-22T15:07:41Z")

</div>

Hi,  
I am fairely new to the elastic stack  
I am using filebeat to pull date from a Rest Api and push it to elastic.  
my configuration file is as follows:

```auto
# ============================== Filebeat inputs ===============================

  - type: httpjson
    interval: 15s
    request.url: https://api.tmes.trendmicro.eu/api/v1/log/mailtrackinglog?type=blocked_traffic&limit=1
    request.transforms:
      - set:
          target: header.Authorization
          value: 'Basic XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
    response.pagination:
      - set:
          target: url.value
          value: https://api.tmes.trendmicro.eu/api/v1/log/mailtrackinglog?type=blocked_traffic&limit=1&token
      - set:
          target: url.params.token
          value: '[[.last_response.body.nextToken]]' # Accessing nextToken here
    processors:
      - decode_json_fields:
          fields: ["message"]
          target: "json"
          document_id: ~
    enabled: true
# ======================= Elasticsearch template setting =======================
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.ilm.enabled: false
setup.template.name: "ecs-MyOrg-filebeat"
setup.template.pattern: "ecs-MyOrg-filebeat*"
setup.template.enabled: true
# =================================== Kibana ===================================

setup.kibana:

  host: "https://11.11.11.11:5601"
  username: "elastic"
  password: "elastic_pass"
  ssl.verification_mode: none
# ---------------------------- Elasticsearch Output ----------------------------

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.0.0.0:9200"]
  action: "create"
  pipeline: "set_document_id"
  index: "ecs-MyOrg-filebeat-trendmicro"
  indicies :
   - index : "ecs-MyOrg-filebeat-trendmicro"
  protocol: "https"
  ssl.enabled: true
  ssl.verification_mode: none
  username: "beats"
  password: "Beats_pass"
  allow_older_versions: true

```

the events go through this pipeline to make sure the document ID is set as mail ID (which should be unique):

```auto
{
  "set_document_id" : {
    "processors" : [
      {
        "script" : {
          "lang" : "painless",
          "source" : "if (ctx.json.logs != null && ctx.json.logs.size() > 0) { ctx['document_id'] = ctx.json.logs[0].mailID; ctx['op_type'] = 'create' }"
        }
      }
    ]
  }
}

```

after all that, Is still get duplicated events, I don't have any idea why.

her's an extract

 ![Screenshot 2023-08-22 160433](https://us1.discourse-cdn.com/elastic/original/3X/8/a/8ad126f91dc2f248ca12d2a96ea5a806578fb17b.png)

please help

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 25, 2023, 12:50pm UTC](https://discuss.elastic.co/t/duplicated-events/341389/3 "2023-08-25T12:50:53Z")

</div>

> [@Mohammed\_Amine\_El\_ha](#):
>
> ```auto
> {
> "set_document_id" : {
> "processors" : [
> {
> "script" : {
> "lang" : "painless",
> "source" : "if (ctx.json.logs != null && ctx.json.logs.size() > 0) { ctx['document_id'] = ctx.json.logs[0].mailID; ctx['op_type'] = 'create' }"
> }
> }
> ]
> }
> }
> 
> ```

You need to set the `_id` field, this pipeline does not set the `_id` field.

Do you have any other ingest pipeline that sets the `_id` field with the value of the `document_id` field? If not, you need to create.

---

<div class="post-metadata">

**Author:** ![Mohammed\_Amine\_El\_ha](https://avatars.discourse-cdn.com/v4/letter/m/bbe5ce/32.png) [@Mohammed\_Amine\_El\_ha](https://discuss.elastic.co/u/Mohammed_Amine_El_ha)\
**Post date:** [August 25, 2023, 4:26pm UTC](https://discuss.elastic.co/t/duplicated-events/341389/4 "2023-08-25T16:26:56Z")

</div>

thanks A LOT,  
the doc wasn't very clear about that.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 22, 2023, 4:27pm UTC](https://discuss.elastic.co/t/duplicated-events/341389/5 "2023-09-22T16:27:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.