# Huge json data, how to stream records fast to filter and to output
**URL:** https://discuss.elastic.co/t/huge-json-data-how-to-stream-records-fast-to-filter-and-to-output/216543
**Category:** Logstash
**Created:** [January 25, 2020, 8:48pm UTC](https://discuss.elastic.co/t/huge-json-data-how-to-stream-records-fast-to-filter-and-to-output/216543 "2020-01-25T20:48:29Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@rkhapre](https://discuss.elastic.co/u/rkhapre)
#### Post date: [January 25, 2020, 8:48pm UTC](https://discuss.elastic.co/t/huge-json-data-how-to-stream-records-fast-to-filter-and-to-output/216543/1 "2020-01-25T20:48:29Z")
Hi All
I have a json data of overall 50, 000 records coming from http input plugin.
When i get the overall response i get it in the column/event : message ( all 50 K records)
Now before leaving to filter i want to process it in the batches of 5k
Like 5K in filter and 5K goes to output
then other 5K processed in filter and then to output...
Currently it process 50K in input, then another 50 K processing in filter and finally reach to output.
How should i make it work like jdbc or elastic input plugin with pagination. I am ready to write any ruby code, but is this something possible?
Thanks
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 26, 2020, 12:10am UTC](https://discuss.elastic.co/t/huge-json-data-how-to-stream-records-fast-to-filter-and-to-output/216543/2 "2020-01-26T00:10:23Z")
> [@rkhapre](#):
>
> is this something possible?
Not with the http input. The code [receives](https://github.com/logstash-plugins/logstash-filter-http/blob/d1776475138dea46a48dac54ab7b4e28caebfad9/lib/logstash/filters/http.rb#L76) the whole response as one unit from the manticore library, and then the entire JSON is [parsed](https://github.com/logstash-plugins/logstash-filter-http/blob/d1776475138dea46a48dac54ab7b4e28caebfad9/lib/logstash/filters/http.rb#L105) into a single object.
---
### Author:  [@rkhapre](https://discuss.elastic.co/u/rkhapre)
#### Post date: [January 26, 2020, 5:23am UTC](https://discuss.elastic.co/t/huge-json-data-how-to-stream-records-fast-to-filter-and-to-output/216543/3 "2020-01-26T05:23:27Z")
I see this is same with http\_poller
Even if i go and develop a simple ruby NET::Http input plugin then also same thing will happen
How can I convert the single message to multiline message in the input itself.
In any case any pipeline will not be able to handle the huge volume of records and then there will be no meaning of persistent queue.
Any good solution to deal with high volume API response
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 23, 2020, 5:32am UTC](https://discuss.elastic.co/t/huge-json-data-how-to-stream-records-fast-to-filter-and-to-output/216543/4 "2020-02-23T05:32:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.