Hi @marc.guasch
Thank you for answering so quickly.
I did many tries because I haven't correctly understood the way the httpjson module handles json arrays (cf a previous topic of mine).
Below is my httpjson config with 3 different attempts (all failed):
- type: httpjson
config_version: 2
interval: 5m
request.url: https://domain.tld/api/v1/...
# FIRST TRY
# response.transforms:
# - set:
# target: body.requested_at
# value: '[[.cursor.requested_at]]'
# default: "[[now]]"
# cursor:
# requested_at:
# value: '[[.first_event.body.requested_at]]'
# default: "[[now]]"
# SECOND TRY
# response.transforms:
# - set:
# target: body.requested_at
# value: '[[.first_event.body.requested_at]]'
# default: "[[now]]"
processors:
- decode_json_fields:
fields: ["message"]
target: "stats"
- drop_fields:
fields: ["message", "stats.rl", "stats.rl_scope", "stats.is_relayed", "stats.pushover_active", "stats.last_pop3_login", "stats.attributes", "stats.relayhost"]
# THIRD TRY
# - add_id:
# target_field: "stats.batch.id"
Both first and second attempts gave a different timestamp to each json object/response/document. And third attempt gave a different uid to each json object/response/document.
A classic response from the server is :
[
{
"max_new_quota": 10737418240,
"username": "info@doman3.tld",
"rl": false,
"is_relayed": 0,
"name": "Full name",
"active": "1",
"domain": "doman3.tld",
"local_part": "info",
"messages": 152,
...
},
{
"max_new_quota": 10737418240,
"username": "test@doman2.tld",
"rl": false,
"is_relayed": 0,
"name": "Full name",
"active": "1",
"domain": "doman2.tld",
"local_part": "test",
"messages": 456,
...
},
...
]
And this response would create these two documents in ES:
{
"_index": "stats-filebeat-2022.01",
"_type": "_doc",
"_id": "l_sGSn4BtCr2vRGF9Z4J",
"_score": 1,
"_source": {
"@timestamp": "2022-01-11T16:44:36.376Z",
"host": ...,
"agent": ...,
"event": {
"created": "2022-01-11T16:44:36.376Z"
},
"ecs": {
"version": "1.12.0"
},
"input": {
"type": "httpjson"
},
"stats": {
"active": 1,
"domain": "doman3.tld",
"name": "Full name",
"local_part": "info",
"max_new_quota": 10737418240,
"username": "info@doman3.tld",
"messages": 152
}
},
"fields": ...
},
{
"_index": "stats-filebeat-2022.01",
"_type": "_doc",
"_id": "BytMTX4BtCr2vRGFqsuZ",
"_score": 1,
"_source": {
"@timestamp": "2022-01-11T16:44:36.377Z",
"host": ...,
"agent": ...,
"event": {
"created": "2022-01-11T16:44:36.377Z"
},
"ecs": {
"version": "1.12.0"
},
"input": {
"type": "httpjson"
},
"stats": {
"active": 1,
"domain": "doman2.tld",
"name": "Full name",
"local_part": "test",
"max_new_quota": 10737418240,
"username": "test@doman2.tld",
"messages": 456
}
},
"fields": ...
}
I did try your config but there is a difference of 3sec between the first and last document, same as my tries.
Is there something I misunderstand?
Thanks again
Thomas