Where to have logic to run everytime a new event occurs

Elastic Stack Logstash
#1

Hi,

I'm writing a new input plugin. I have a need to run some logic everytime a new event is captured by Logstash.
Based on the outcome of the logic I want to decide whether the event will be further processed or will it be dropped.

Ex: say with file input plugin - I want to do some validation on the new file and based on it I want to allow it to be processed or terminated.

For this where do I write my logic?? I tried in the run(queue) method but seems like it is only run at the initiation of the input plugin.

Shabir

#2

Are you sure you want to do this in an input plugin? It sounds like you want to write a filter plugin that e.g. sets a tag if the event matches your set of conditions. Then you can add a conditional that checks for that tag and drops the event if there's a match. You could drop the event from your filter too but that's less flexible.

#3

Hi,

To be more specific what I'm trying to do is to validate every new file that's dropped for processing by the logstash-input-file plugin.

How I do this is I calculate the MD5 of the new file and call an endpoint to check whether it's a valid checksum. For this, I have added a new param to the input plugin called auth-endpoint.

So whenever a new file is detected, the MD5 digest of it is calculated and the endpoint above is called with this MD5 to check if it's a valid file.

I have explained what I have done here: https://github.com/elastic/logstash/issues/6895

However, I need more advice as to what I have done is correct? Is it possible to achieve what I have described with filters, the way you had explained? If so can you give me a more concrete example?

I am very new to logstash.

Shabir

#4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.