Saving request.url variable in filebeat httpjson

Elastic Stack Beats
1

Good day! The question arose, is it possible to save the last received value of the request.url parameter and then read from it upon request?
I use httpjson as an input module with the following settings:

# ============================== Filebeat inputs ===============================

filebeat.inputs:

- type: httpjson
  config_version: 2
  interval: 1m
  request.url: https://mysite.com/api/v2/data/updated?updateID=12345
  request.method: GET
  auth.basic.enabled: true
  auth.basic.user: login
  auth.basic.password: password
  request.transforms:
    - set:
        target: url.params.updateID
        value: '[[.last_response.body.updateID]]'

In response, I receive data, from which, using request.transforms, the updateID parameter is taken from the last received data and substituted into the original request, which allows me to receive only data starting from the last updateID. Everything works correctly, as long as there is no need to restart filebeat. Data with ID 12345 is re-received and only then actual data. Is it possible to save the received updateID value from the last request somewhere and read from it in the future?

2

Yes using the cursor. See HTTP JSON input | Filebeat Reference [7.15] | Elastic

1 Like
3

Thank you, it works!
Do you know how to clear state of the cursor, to process again from the beginning?
Where it keeps it condition?

4

What do u mean exactly? When u set the cursor it updates on every run based off the value you set and u can set multiple cursor variables. What would be the use case to clear it? I don't think there is a way to do that.

5

If I get some data using httpjson-cursor and then delete index, containing data from it I can't get this data again. Maybe I doing something wrong?

6

Ohh so you're atalking about clearing it manually?? I suspect its stored in the registry file but not 100%, you'll have to try that and see.

7

yes, you're right, I want to clear it manually.
In registry directory (/var/lib/filebeat/registry/filebeat) there is only log.json and meta.json and it doesn't look like zeroing one of them will give the result

8

Based on the code it looks like the cursor is just stored in memory so idk the best way to reset it.

9

Ok, thank you!

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.