I'm writing a new input plugin. I have a need to run some logic everytime a new event is captured by Logstash.
Based on the outcome of the logic I want to decide whether the event will be further processed or will it be dropped.
Ex: say with file input plugin - I want to do some validation on the new file and based on it I want to allow it to be processed or terminated.
For this where do I write my logic?? I tried in the run(queue) method but seems like it is only run at the initiation of the input plugin.
Are you sure you want to do this in an input plugin? It sounds like you want to write a filter plugin that e.g. sets a tag if the event matches your set of conditions. Then you can add a conditional that checks for that tag and drops the event if there's a match. You could drop the event from your filter too but that's less flexible.
To be more specific what I'm trying to do is to validate every new file that's dropped for processing by the logstash-input-file plugin.
How I do this is I calculate the MD5 of the new file and call an endpoint to check whether it's a valid checksum. For this, I have added a new param to the input plugin called auth-endpoint.
So whenever a new file is detected, the MD5 digest of it is calculated and the endpoint above is called with this MD5 to check if it's a valid file.
However, I need more advice as to what I have done is correct? Is it possible to achieve what I have described with filters, the way you had explained? If so can you give me a more concrete example?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.