Stop logstash after processing the file

Elastic Stack Logstash
1

Hi all,

I have a shell script which will be triggered when ever a new file is arrived into a folder called inbound.
logstash process needs to be started to read and load the file contents into elastic.

I am able to do the above steps but the logstash process is still in waiting mode. How can i safely exit the logstash process to mark the file loading is completed.

Regards
KK

2

This has been covered here many times before. In short you have two options:

  • Monitor the sincedb file to check when Logstash has reached EOF, then kill Logstash.
  • Use the stdin input instead. Logstash will shut down by itself when stdin has been closed and all inputs has been processed. On the flip side, it Logstash dies for whatever reason you don't know how much it has processed.

But why do you need to start and stop Logstash on demand? Can't you let it run all the time and process files as they arrive?

3

THanks for your response. the reason why we want to stop and start te logstash on demand because, we receive the file from other system only once in a day. so, dont want to keep running this process for ever.

secondly, we want to move the data file to archives folder for audit purpose. so we have to know if logstash has completed the processing of the file or not.

Can you please provide any example script code for the 2 solutions that you have provided. thanks much.

4

I don't have time to provide complete examples, but I can answer specific questions you might have. Also, see http://stackoverflow.com/questions/33143437/logstash-parsing-progress-bar for details on how to know when Logstash is done processing a file.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.