Whether log stash parsed log file completely


(Siddhant Agarwal) #1

I want to run a script when log stash has finished parsing log file completely. Can anyone suggest a way to do that?


(Ed) #2

Do you have an example of your current configuration?

Logstash if using the "file" input is built for continuous reading of a file "Like tail" there is really no time it stops reading a file. It is not really set up as an Event handler.as your asking for.

You can run a script on the the contents of the last line read. If you have one ( or } ) but even that is not an event, It would kick off the script every time it occurred not just EOF


(Siddhant Agarwal) #3

I was using the file input and was using exec output to script but it was executing at every event as mentioned by you. So I solved the problem by using stdin in input and directing log file. It finishes after parsing and then I ran the script using bash script.
Thanks for your reply.


(Siddhant Agarwal) #4

But this will not work if I have multiple log files or a directory of log files.


(Mark Walkom) #5

There's no real good way to do this.

Why are you trying to do this?


(Siddhant Agarwal) #6

I am using python API for elastic search to do some operations on various fields and putting results into elastic search. So as I want to automate this working I want to execute it after log stash has finished parsing log file/files.


(Magnus Bäck) #7

So... iterate over the files and invoke Logstash multiple times? Or concatenate the files?

cat *.log | logstash ...

(Siddhant Agarwal) #8

Thanks for the reply @magnusbaeck.
I will do following:
cat path_to_log_files/*.log > temp.log
bin/logstash -f example.conf < temp.log


(Magnus Bäck) #9

Unless you need to support interruptions and recovering from them there's no point with the temporary file. Just pipe the output of cat straight to Logstash as I showed you.


(system) #10