# Using recursion for paging?

**URL:** <https://discuss.elastic.co/t/using-recursion-for-paging/282079>\
**Category:** Logstash\
**Created:** [August 20, 2021, 12:55pm UTC](https://discuss.elastic.co/t/using-recursion-for-paging/282079 "2021-08-20T12:55:09Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jasenj1](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@jasenj1](https://discuss.elastic.co/u/jasenj1)\
**Post date:** [August 20, 2021, 12:55pm UTC](https://discuss.elastic.co/t/using-recursion-for-paging/282079/1 "2021-08-20T12:55:10Z")

</div>

Given a web service that supports a typical paging construct of specifying the record to start from and a page size, is using the "pipeline" input and output to recursively call a config file a viable option? Or will it result in a memory bomb?

Here is how I have a recursive algorithm currently implemented:

1. An "initial" config file is set to run on a schedule. It creates an event with some initial values (start\_at\_record = 0, page\_size = 10), and outputs to another pipeline, "worker".
2. "worker" uses the HTTP filter to retrieve a batch of data from the web service using start\_at\_record, and page\_size.
3. One of the resulting records is marked "keep\_going". (I use a bit of Ruby to do this.)
4. The "split" filter is used to break the set of records into individual records for processing.
5. When the "keep\_going" flag is detected, start\_at\_record is incremented by page\_size.
6. The 'output' section has an if statement so that wen 'keep\_going' is detected, 'pipeline' is used to call the "worker" config again.
7. Goto 2.
8. Eventually, there are no more records returned, no record gets marked "keep\_going" and everything falls out the bottom.

Is this clever? Or courting disaster?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 17, 2021, 12:55pm UTC](https://discuss.elastic.co/t/using-recursion-for-paging/282079/2 "2021-09-17T12:55:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.