I want to run a script when log stash has finished parsing log file completely. Can anyone suggest a way to do that?
Do you have an example of your current configuration?
Logstash if using the "file" input is built for continuous reading of a file "Like tail" there is really no time it stops reading a file. It is not really set up as an Event handler.as your asking for.
You can run a script on the the contents of the last line read. If you have one ( or } ) but even that is not an event, It would kick off the script every time it occurred not just EOF
I was using the file input and was using exec output to script but it was executing at every event as mentioned by you. So I solved the problem by using stdin in input and directing log file. It finishes after parsing and then I ran the script using bash script.
Thanks for your reply.
But this will not work if I have multiple log files or a directory of log files.
There's no real good way to do this.
Why are you trying to do this?
I am using python API for elastic search to do some operations on various fields and putting results into elastic search. So as I want to automate this working I want to execute it after log stash has finished parsing log file/files.
So... iterate over the files and invoke Logstash multiple times? Or concatenate the files?
cat *.log | logstash ...
Thanks for the reply @magnusbaeck.
I will do following:
cat path_to_log_files/*.log > temp.log
bin/logstash -f example.conf < temp.log
Unless you need to support interruptions and recovering from them there's no point with the temporary file. Just pipe the output of cat straight to Logstash as I showed you.