I have a json data of overall 50, 000 records coming from http input plugin.
When i get the overall response i get it in the column/event : message ( all 50 K records)
Now before leaving to filter i want to process it in the batches of 5k
Like 5K in filter and 5K goes to output
then other 5K processed in filter and then to output...
Currently it process 50K in input, then another 50 K processing in filter and finally reach to output.
How should i make it work like jdbc or elastic input plugin with pagination. I am ready to write any ruby code, but is this something possible?
Not with the http input. The code receives the whole response as one unit from the manticore library, and then the entire JSON is parsed into a single object.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.