HTTPJSON input : get only new records

Elastic Stack Beats
1

hi everyone!

im trying to get audit logs from a cloud api, the logs on the api are always the same unless there is a new login happening.

while using the httpjson input im always getting all data again, creating multiple duplicate records, which is expected...i've been looking at request.transforms but i cant manage to get it working

  request.transforms:
    - set:
        target: last_response_body
        value: '[[.last_response.body.id]]'

this does nothing on my case... im a bit lost.

i would like to only get new records = newer ID's on the api based on the last response we got from it

example of api records

[
{
"id": 4603,
"time": "2022-08-24T15:11:21.181120-07:00",
"body": {
"user": "xxx@xxx.org",
"details": "Login succeeded from 1.1.1.1 with identity provider 'something'. Granted privilege 'writelevel: access_admin' based on OIDC attribute 'custom:write_level', value 'access_admin'. Granted privilege 'packetslevel: full_with_keys' based on OIDC attribute 'custom:packets_level', value 'full_with_keys'. Granted privilege 'detectionsaccesslevel: full' based on OIDC attribute 'custom:detections_level', value 'full'.",
"facility": "OIDC",
"priority": 6,
"operation": "Login"
},
"occur_time": 1661379081181
},
{
"id": 4602,
"time": "2022-08-24T15:10:57.636993-07:00",
"body": {
"user": "xxx@xxx.org",
"details": "Login succeeded from 1.1.1.1 with identity provider 'something'. Granted privilege 'writelevel: access_admin' based on OIDC attribute 'custom:write_level', value 'access_admin'. Granted privilege 'packetslevel: full_with_keys' based on OIDC attribute 'custom:packets_level', value 'full_with_keys'. Granted privilege 'detectionsaccesslevel: full' based on OIDC attribute 'custom:detections_level', value 'full'.",
"facility": "OIDC",
"priority": 6,
"operation": "Login"
},
"occur_time": 1661379057637
}
]

so, after the first run, i would only like to send if record is 4604, and so on (whatever id was last, since the last time filebeat went out to get the api records) i believe i should also use the cursor option? ...

appreciate if anyone could lend a hand, would be highly appreciated!!

Daniel

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.