Duplicated events

[Mohammed_Amine_El_ha (Mamoun)]

Hi,
I am fairely new to the elastic stack
I am using filebeat to pull date from a Rest Api and push it to elastic.
my configuration file is as follows:

# ============================== Filebeat inputs ===============================

  - type: httpjson
    interval: 15s
    request.url: https://api.tmes.trendmicro.eu/api/v1/log/mailtrackinglog?type=blocked_traffic&limit=1
    request.transforms:
      - set:
          target: header.Authorization
          value: 'Basic XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
    response.pagination:
      - set:
          target: url.value
          value: https://api.tmes.trendmicro.eu/api/v1/log/mailtrackinglog?type=blocked_traffic&limit=1&token
      - set:
          target: url.params.token
          value: '[[.last_response.body.nextToken]]'  # Accessing nextToken here
    processors:
      - decode_json_fields:
          fields: ["message"]
          target: "json"
          document_id: ~
    enabled: true
# ======================= Elasticsearch template setting =======================
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.ilm.enabled: false
setup.template.name: "ecs-MyOrg-filebeat"
setup.template.pattern: "ecs-MyOrg-filebeat*"
setup.template.enabled: true
# =================================== Kibana ===================================

setup.kibana:

  host: "https://11.11.11.11:5601"
  username: "elastic"
  password: "elastic_pass"
  ssl.verification_mode: none
# ---------------------------- Elasticsearch Output ----------------------------

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.0.0.0:9200"]
  action: "create"
  pipeline: "set_document_id"
  index: "ecs-MyOrg-filebeat-trendmicro"
  indicies :
   - index : "ecs-MyOrg-filebeat-trendmicro"
  protocol: "https"
  ssl.enabled: true
  ssl.verification_mode: none
  username: "beats"
  password: "Beats_pass"
  allow_older_versions: true

the events go through this pipeline to make sure the document ID is set as mail ID (which should be unique):

{
  "set_document_id" : {
    "processors" : [
      {
        "script" : {
          "lang" : "painless",
          "source" : "if (ctx.json.logs != null && ctx.json.logs.size() > 0) { ctx['document_id'] = ctx.json.logs[0].mailID; ctx['op_type'] = 'create' }"
        }
      }
    ]
  }
}

after all that, Is still get duplicated events, I don't have any idea why.

her's an extract

please help

[leandrojmp (Leandro Pereira)]

@Mohammed_Amine_El_ha

{
  "set_document_id" : {
    "processors" : [
      {
        "script" : {
          "lang" : "painless",
          "source" : "if (ctx.json.logs != null && ctx.json.logs.size() > 0) { ctx['document_id'] = ctx.json.logs[0].mailID; ctx['op_type'] = 'create' }"
        }
      }
    ]
  }
}

You need to set the _id field, this pipeline does not set the _id field.

Do you have any other ingest pipeline that sets the _id field with the value of the document_id field? If not, you need to create.

[Mohammed_Amine_El_ha (Mamoun)]

thanks A LOT,
the doc wasn't very clear about that.