How to call python script from Logstash to process (partialy) logs


I have to process parts of the logging information using python script (python beautifulsoup and other useful stuff which I do not want to reimnplement in Ruby).

What is the simplest way to do this?

As of now I have found two ways:

  1. exec output plugin. In this case, my python script is launched after elasticsearch output and I can make additional processing and update event in Elasticsearch at the end (sending all the required ids needed to the script through cmdline)
  2. Create my own filter plugin for logstash and call python script from that plugin

It looks like second option is better in terms of performance (no need to do insert/update to Elastic, I will insert data to the database only once) but a little bit more complicated than the first option.

Do you have any other ideas?

Thanks in advance!

Your second option is way better than the first. You really should scrap the first idea.

Another option would be to run two Logstash instances where the first posts to a broker (e.g. Redis or RabbitMQ) that your Python script consumes from and posts back to the second Logstash instance which posts to Elasticsearch (you could do the same thing with a single instance).


Thanks, Magnus. I have found the third option - called python script from Logstash filtering pipeline using "ruby" plugin. Documented my solution here:

It was a little bit faster than to create my own plugin with similar functionality (calling external command from it and analyzing its output).


It seems the code content in the link provided by you is not working now.

I tried using event[] syntax in ruby and it failed. I think the method to read & Store the events from outside ruby code has changed. Any idea how to access the events from the message ?


This was a change in Logstash 5.0, documented here:

Thanks. It worked.