Ingest Node Processor for cumulative count of records

[RohanKumbhar (R01K)]

Hi Team,

Is there any processor in ingest pipeline which will give me the cumulative count the records processed through the ingest node?

for example there are 10 records processed by ingest node then cumulative count should be 10 and for next record it should be 11.

Thanks,
Rohan

[dadoonet (David Pilato)]

No. It doesn’t exist and I don’t think it can exist TBH.

[RohanKumbhar (R01K)]

Thanks @dadoonet for your response!! Just a suggestion; can we add processor "metrics processor" similar to metrics filter plugin of logstash ? which help to get the required details.

[dadoonet (David Pilato)]

I don't know how the metrics filter plugin works behind the scene but I believe it is storing "locally" on the Logstash instance the "counters". Which means to me that it can not be distributed within multiple Logstash instances.

With that said, ingest node is distributed by nature. Which means that Node 1 will handle a 1st request, then may be Node 2, then may be Node3... And you can easily understand what is going to happen in that case.

If Logstash could work for you, I'd go the logstash way.
If you really want to do it in Ingest, then you need to:

• write your own ingest plugin
• make sure that only one node is set to "node.ingest": true and other nodes are set to false.

My 2 cents.

[RohanKumbhar (R01K)]

sure Thanks @dadoonet !!