# How to do batched fetching with jdbc\_streaming?

**URL:** <https://discuss.elastic.co/t/how-to-do-batched-fetching-with-jdbc-streaming/367265>\
**Category:** Logstash\
**Created:** [September 27, 2024, 9:04pm UTC](https://discuss.elastic.co/t/how-to-do-batched-fetching-with-jdbc-streaming/367265 "2024-09-27T21:04:53Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mxu](https://avatars.discourse-cdn.com/v4/letter/m/90ced4/32.png) [@mxu](https://discuss.elastic.co/u/mxu)\
**Post date:** [September 27, 2024, 9:04pm UTC](https://discuss.elastic.co/t/how-to-do-batched-fetching-with-jdbc-streaming/367265/1 "2024-09-27T21:04:53Z")

</div>

I have to use jdbc\_streaming to get data from postgres database. With large number of records, logstash runs OOM. Instead of increasing heap size, I'd like to batched read as jdbc input would do.  
have been trying different ways (using OFFSET and LIMIT for query and write offset to a track\_offset file). Cannot get it working.  
Anyone has experience about this?  
TIA

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 27, 2024, 9:58pm UTC](https://discuss.elastic.co/t/how-to-do-batched-fetching-with-jdbc-streaming/367265/2 "2024-09-27T21:58:30Z")

</div>

> [@mxu](#):
>
> I have to use jdbc\_streaming to get data from postgres database. With large number of records, logstash runs OOM. Instead of increasing heap size, I'd like to batched read as jdbc input would do.

The jdbc\_streaming filter returns the entire result set in a single event as an array of hashes. If that doesn't fit into the available memory then it will indeed run the JVM OOM. If you want one event per DB row then currently you have to use a split filter after the jdbc\_streaming filter.

It would be possible to re-architect the jdbc\_streaming filter to have it use paging and do the splits itself but that would be quite a large project.

---

<div class="post-metadata">

**Author:** ![mxu](https://avatars.discourse-cdn.com/v4/letter/m/90ced4/32.png) [@mxu](https://discuss.elastic.co/u/mxu)\
**Post date:** [September 29, 2024, 12:14am UTC](https://discuss.elastic.co/t/how-to-do-batched-fetching-with-jdbc-streaming/367265/3 "2024-09-29T00:14:00Z")

</div>

Thank you, Badger.  
The jdbc\_streaming query could return as many as 5m rows. I have following split script after the query.

```auto
def register(params)
    @field = params['field']
    @target = params['target']
end

def filter(event)
    data = event.get(@field)
    event.remove(@field)
    a = []
    data.each { |x|
        e = event.clone
        e.set(@target, x)
        a << e
    }
    a
end

```

```auto
type or paste code here

```