# đź“Ś \[Filebeat HTTPJSON\] Preserve Last Page Cursor for Next Run When Pagination Ends **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [May 25, 2025, 12:35pm UTC](https://discuss.elastic.co/t/filebeat-httpjson-preserve-last-page-cursor-for-next-run-when-pagination-ends/378501 "2025-05-25T12:35:11Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![mehrad\_ghalibafi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehrad_ghalibafi/32/142941_2.png) [@mehrad\_ghalibafi](https://discuss.elastic.co/u/mehrad_ghalibafi)\ **Post date:** [May 25, 2025, 12:35pm UTC](https://discuss.elastic.co/t/filebeat-httpjson-preserve-last-page-cursor-for-next-run-when-pagination-ends/378501/1 "2025-05-25T12:35:11Z")
**Hi Elastic Team,** I'm using **Filebeat’s `httpjson` input** to integrate with a paginated API that uses a `next` field for pagination. Here's what I have working so far: ### ✅ Current Working Setup - `response.pagination` is configured to follow the `next` page value but doesn't wotk: ```auto "response.pagination": [ { "set": { "target": "url.value", "value": "[[.last_response.body.next]]" }, "if": "[[.last_response.body.next]] != null" }, { "stop": {}, "if": "[[.last_response.body.next]] == null" } ] ``` - This is working fine: ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b4afe10893b952afcefe97af30718f2212c64934.png) - `response.split` works correctly on `body.results`. - I’m using `document_id: [[.id]]` to avoid duplicates. ### ❌ The Problem After the last page is reached (`"next": null`), the pipeline **stops as expected**. But on the **next run** , Filebeat **starts again from the first page** because there’s no mechanism to **store and reuse the last page URL**. ### 🎯 What I Want I want Filebeat to: - **Store the last known page URL** (before reaching `null`), and - **Start from that stored URL** in the **next interval run** (rather than going back to the first page). * * * ### 🔍 Attempted Solutions - I tried using a second `set` on `.last_response.body.previous` if `next` is `null` — but that’s only for the current run and doesn’t persist. - I looked at `custom request cursor`, but there’s no way to store a fallback value in it on `stop`. * * * ### 🙏 Feature Request / Help Needed Is there a way to: - **Persist the cursor** (URL or ID) when pagination ends? - **Set a fallback cursor** on `stop` condition? - Or enhance `cursor` handling to **cache the last `next` value** across runs? Thanks for any insights or guidance! ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [May 29, 2025, 1:22am UTC](https://discuss.elastic.co/t/filebeat-httpjson-preserve-last-page-cursor-for-next-run-when-pagination-ends/378501/3 "2025-05-29T01:22:05Z")
We recommend the CEL input (\_ Custom API using Common Expression Language\_) for all new API integrations because it is more flexible. In order to persist state between intervals you will need to configure the "custom request cursor" section ([HTTP JSON input | Filebeat Reference [8.18] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html#cursor)). It might be helpful to look at some of the many examples in our integrations repo: [Code search results · GitHub](https://github.com/search?q=repo%3Aelastic%2Fintegrations%20path%3A**%2Fhttpjson.yml.hbs%20%2Fcursor%3A%2F&type=code) ---
**Author:** ![mehrad\_ghalibafi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehrad_ghalibafi/32/142941_2.png) [@mehrad\_ghalibafi](https://discuss.elastic.co/u/mehrad_ghalibafi)\ **Post date:** [June 1, 2025, 8:28am UTC](https://discuss.elastic.co/t/filebeat-httpjson-preserve-last-page-cursor-for-next-run-when-pagination-ends/378501/4 "2025-06-01T08:28:23Z")
I'm configuring a Custom API integration using Common Expression Language (CEL) in Elastic, aiming to paginate through API responses until the `next` field is `null` then save state with `previous` page for next run. However, I'm encountering the following error during program compilation: ```auto Failed failed to check program: failed compilation: ERROR: :1:11: Syntax error: token recognition error at: '= ' | respBytes = get(state.page).Body | ..........^ ERROR: :1:13: Syntax error: mismatched input 'get' expecting | respBytes = get(state.page).Body | ............^ ERROR: :2:9: Syntax error: token recognition error at: '= ' | decoded = body.decode_json(respBytes) | ........^ accessing config:contentReference[oaicite:14]{index=14} ``` Here's the API response I'm working with: ```auto { "count": 243112, "next": "http://192.168.1.1:3400/feed/indicator/?page=2", "previous": null, "results": [ {}, {} ] } ``` And this is the CEL program I'm using which is generated by AI: ```auto respBytes := get(state.page).Body decoded := body.decode_json(respBytes) { events: decoded.results, state: { page: if decoded.next != null && decoded.next != "" then decoded.next else state.page } } ``` I'm not sure what's causing these syntax errors. Could someone help me identify and resolve the issue? Thank you in advance for your assistance! ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [June 10, 2025, 6:51pm UTC](https://discuss.elastic.co/t/filebeat-httpjson-preserve-last-page-cursor-for-next-run-when-pagination-ends/378501/5 "2025-06-10T18:51:04Z")
Do you have a specification for this API that you can share? Here's something that will fetch the data and turn each `result` into an event. It is adapted from an existing integration (linked below). But without knowing more about the API behavior, I cannot say if this is correct. ```auto state.with( get(state.?next.orValue(state.url)).as(resp, (resp.StatusCode == 200) ? bytes(resp.Body).decode_json().as(body, { "events": body.results.map(e, { "message": e.encode_json(), } ), ?"next": body.?next, "want_more": has(body.next), } ) : { "events": { "error": { "code": string(resp.StatusCode), "id": string(resp.Status), "message": "GET:" + ( (size(resp.Body) != 0) ? string(resp.Body) : string(resp.Status) + " (" + string(resp.StatusCode) + ")" ), }, }, "want_more": false, } ) ) ``` References - [integrations/packages/authentik/data\_stream/event/agent/stream/cel.yml.hbs at ef0d11752ad849c450370f7b2c5405454e3abd30 · elastic/integrations · GitHub](https://github.com/elastic/integrations/blob/ef0d11752ad849c450370f7b2c5405454e3abd30/packages/authentik/data_stream/event/agent/stream/cel.yml.hbs#L24-L61)