HTTPJSON input : share request identifier among events

Elastic Stack Beats
3

Hi @marc.guasch

Thank you for answering so quickly.

I did many tries because I haven't correctly understood the way the httpjson module handles json arrays (cf a previous topic of mine).

Below is my httpjson config with 3 different attempts (all failed):

- type: httpjson
  config_version: 2
  interval: 5m
  request.url: https://domain.tld/api/v1/...
# FIRST TRY
#  response.transforms:
#    - set:
#        target: body.requested_at
#        value: '[[.cursor.requested_at]]'
#        default: "[[now]]"
#  cursor:
#    requested_at:
#      value: '[[.first_event.body.requested_at]]'
#      default: "[[now]]"
# SECOND TRY
#  response.transforms:
#    - set:
#        target: body.requested_at
#        value: '[[.first_event.body.requested_at]]'
#        default: "[[now]]"
  processors:
    - decode_json_fields:
        fields: ["message"]
        target: "stats"
    - drop_fields:
        fields: ["message", "stats.rl", "stats.rl_scope", "stats.is_relayed", "stats.pushover_active", "stats.last_pop3_login", "stats.attributes", "stats.relayhost"]
# THIRD TRY
#    - add_id: 
#        target_field: "stats.batch.id"

Both first and second attempts gave a different timestamp to each json object/response/document. And third attempt gave a different uid to each json object/response/document.

A classic response from the server is :

[
  {
    "max_new_quota": 10737418240,
    "username": "info@doman3.tld",
    "rl": false,
    "is_relayed": 0,
    "name": "Full name",
    "active": "1",
    "domain": "doman3.tld",
    "local_part": "info",
    "messages": 152,
    ...
  },
  {
    "max_new_quota": 10737418240,
    "username": "test@doman2.tld",
    "rl": false,
    "is_relayed": 0,
    "name": "Full name",
    "active": "1",
    "domain": "doman2.tld",
    "local_part": "test",
    "messages": 456,
    ...
  },
  ...
]

And this response would create these two documents in ES:

{
  "_index": "stats-filebeat-2022.01",
  "_type": "_doc",
  "_id": "l_sGSn4BtCr2vRGF9Z4J",
  "_score": 1,
  "_source": {
    "@timestamp": "2022-01-11T16:44:36.376Z",
    "host": ...,
    "agent": ...,
    "event": {
      "created": "2022-01-11T16:44:36.376Z"
    },
    "ecs": {
      "version": "1.12.0"
    },
    "input": {
      "type": "httpjson"
    },
    "stats": {
      "active": 1,
      "domain": "doman3.tld",
      "name": "Full name",
      "local_part": "info",
      "max_new_quota": 10737418240,
      "username": "info@doman3.tld",
      "messages": 152
    }
  },
  "fields": ...
},
{
  "_index": "stats-filebeat-2022.01",
  "_type": "_doc",
  "_id": "BytMTX4BtCr2vRGFqsuZ",
  "_score": 1,
  "_source": {
    "@timestamp": "2022-01-11T16:44:36.377Z",
    "host": ...,
    "agent": ...,
    "event": {
      "created": "2022-01-11T16:44:36.377Z"
    },
    "ecs": {
      "version": "1.12.0"
    },
    "input": {
      "type": "httpjson"
    },
    "stats": {
      "active": 1,
      "domain": "doman2.tld",
      "name": "Full name",
      "local_part": "test",
      "max_new_quota": 10737418240,
      "username": "test@doman2.tld",
      "messages": 456
    }
  },
  "fields": ...
}

I did try your config but there is a difference of 3sec between the first and last document, same as my tries.
Is there something I misunderstand?

Thanks again

Thomas