Duplicated events

I am fairely new to the elastic stack
I am using filebeat to pull date from a Rest Api and push it to elastic.
my configuration file is as follows:

# ============================== Filebeat inputs ===============================

  - type: httpjson
    interval: 15s
    request.url: https://api.tmes.trendmicro.eu/api/v1/log/mailtrackinglog?type=blocked_traffic&limit=1
      - set:
          target: header.Authorization
      - set:
          target: url.value
          value: https://api.tmes.trendmicro.eu/api/v1/log/mailtrackinglog?type=blocked_traffic&limit=1&token
      - set:
          target: url.params.token
          value: '[[.last_response.body.nextToken]]'  # Accessing nextToken here
      - decode_json_fields:
          fields: ["message"]
          target: "json"
          document_id: ~
    enabled: true
# ======================= Elasticsearch template setting =======================
  index.number_of_shards: 1
  index.codec: best_compression
setup.ilm.enabled: false
setup.template.name: "ecs-MyOrg-filebeat"
setup.template.pattern: "ecs-MyOrg-filebeat*"
setup.template.enabled: true
# =================================== Kibana ===================================


  host: ""
  username: "elastic"
  password: "elastic_pass"
  ssl.verification_mode: none
# ---------------------------- Elasticsearch Output ----------------------------

  # Array of hosts to connect to.
  hosts: [""]
  action: "create"
  pipeline: "set_document_id"
  index: "ecs-MyOrg-filebeat-trendmicro"
  indicies :
   - index : "ecs-MyOrg-filebeat-trendmicro"
  protocol: "https"
  ssl.enabled: true
  ssl.verification_mode: none
  username: "beats"
  password: "Beats_pass"
  allow_older_versions: true

the events go through this pipeline to make sure the document ID is set as mail ID (which should be unique):

  "set_document_id" : {
    "processors" : [
        "script" : {
          "lang" : "painless",
          "source" : "if (ctx.json.logs != null && ctx.json.logs.size() > 0) { ctx['document_id'] = ctx.json.logs[0].mailID; ctx['op_type'] = 'create' }"

after all that, Is still get duplicated events, I don't have any idea why.

her's an extract

please help

You need to set the _id field, this pipeline does not set the _id field.

Do you have any other ingest pipeline that sets the _id field with the value of the document_id field? If not, you need to create.

thanks A LOT,
the doc wasn't very clear about that.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.