Huge json data, how to stream records fast to filter and to output

[rkhapre (RK)]

Hi All

I have a json data of overall 50, 000 records coming from http input plugin.
When i get the overall response i get it in the column/event : message ( all 50 K records)

Now before leaving to filter i want to process it in the batches of 5k

Like 5K in filter and 5K goes to output
then other 5K processed in filter and then to output...

Currently it process 50K in input, then another 50 K processing in filter and finally reach to output.

How should i make it work like jdbc or elastic input plugin with pagination. I am ready to write any ruby code, but is this something possible?

Thanks

[Badger]

@rkhapre

is this something possible?

Not with the http input. The code receives the whole response as one unit from the manticore library, and then the entire JSON is parsed into a single object.

[rkhapre (RK)]

I see this is same with http_poller
Even if i go and develop a simple ruby NET::Http input plugin then also same thing will happen

How can I convert the single message to multiline message in the input itself.

In any case any pipeline will not be able to handle the huge volume of records and then there will be no meaning of persistent queue.

Any good solution to deal with high volume API response