Reading gzipped files and shutting down Logstash once done

Elastic Stack Logstash
1

Hi all,
My use case is as follows - I need to process gzipped files by demand and I cannot afford having an instance of Logstash continuously running - therefore I must launch an instance, have it process the file, and terminate it once done.

I tried two approaches:

  1. Reading the file with the "file" input plugin. This reads the file well, and I have an indication on when the processing is done by using the "file_completed_action", but there is no option to execute Logstash on a single file, so I have to manage the state of the Logstash process on my own.

  2. Reading the file with
    ./logstash -e 'input { stdin {} } output { ANY_OUTPUT_FILTER }' < /file_path/test.log.gz
    This method will exit after the processing is done, yet reading gzipped files doesn't work at all. I've also tried using the "gzip_lines" codec, which also unfortunately doesn't work.

Does anyone have any other idea/suggestion that would suit this problem?

2

If you are willing to do that then why not do this?

gunzip -c /file_path/test.log.gz | ./logstash -e 'input { stdin {} } output { ANY_OUTPUT_FILTER }'
1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.