Where to have logic to run everytime a new event occurs


I'm writing a new input plugin. I have a need to run some logic everytime a new event is captured by Logstash.
Based on the outcome of the logic I want to decide whether the event will be further processed or will it be dropped.

Ex: say with file input plugin - I want to do some validation on the new file and based on it I want to allow it to be processed or terminated.

For this where do I write my logic?? I tried in the run(queue) method but seems like it is only run at the initiation of the input plugin.


Are you sure you want to do this in an input plugin? It sounds like you want to write a filter plugin that e.g. sets a tag if the event matches your set of conditions. Then you can add a conditional that checks for that tag and drops the event if there's a match. You could drop the event from your filter too but that's less flexible.


To be more specific what I'm trying to do is to validate every new file that's dropped for processing by the logstash-input-file plugin.

How I do this is I calculate the MD5 of the new file and call an endpoint to check whether it's a valid checksum. For this, I have added a new param to the input plugin called auth-endpoint.

So whenever a new file is detected, the MD5 digest of it is calculated and the endpoint above is called with this MD5 to check if it's a valid file.

I have explained what I have done here: https://github.com/elastic/logstash/issues/6895

However, I need more advice as to what I have done is correct? Is it possible to achieve what I have described with filters, the way you had explained? If so can you give me a more concrete example?

I am very new to logstash.


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.