HTTPJSON input : share request identifier among events

Okay I think I know what is going on here.

For the third case, this is expected as the input processors get executed on each published event.

For the first two, when the httpjson input receives an array response it is automatically split, generating one document for each object found in it. This implicit top level split step can't be skipped, and any defined split in the config will behave as if they were nested under it. Seeing the documentation I couldn't find any reference to this behavior, but I created this issue to amend this. Since there is no top level object common to all documents, response.transforms can't be used straight away as I suggested first, since, as you mentioned, it will be executed separately for each of them after the split is done.

As a workaround, and to avoid doing complicated things with the cursor and templates, you can rely on the Date header, which should be part of the server response:

- type: httpjson
  config_version: 2
  interval: 5m
  request.url: https://domain.tld/api/v1/...
  response.transforms:
    - set:
        target: body.requested_at
        value: '[[.last_response.header.Get "Date"]]'
processors:
  - decode_json_fields:
      fields: ["message"]
      target: "stats"
  - drop_fields:
      fields: ["message", "stats.rl", "stats.rl_scope", "stats.is_relayed", "stats.pushover_active", "stats.last_pop3_login", "stats.attributes", "stats.relayhost"]

You can format it as you wish with the parseDate and formatDate template functions.

Hope that helps and makes sense!

1 Like