I have to process parts of the logging information using python script (python beautifulsoup and other useful stuff which I do not want to reimnplement in Ruby).
What is the simplest way to do this?
As of now I have found two ways:
exec output plugin. In this case, my python script is launched after elasticsearch output and I can make additional processing and update event in Elasticsearch at the end (sending all the required ids needed to the script through cmdline)
Create my own filter plugin for logstash and call python script from that plugin
It looks like second option is better in terms of performance (no need to do insert/update to Elastic, I will insert data to the database only once) but a little bit more complicated than the first option.
Your second option is way better than the first. You really should scrap the first idea.
Another option would be to run two Logstash instances where the first posts to a broker (e.g. Redis or RabbitMQ) that your Python script consumes from and posts back to the second Logstash instance which posts to Elasticsearch (you could do the same thing with a single instance).
It seems the code content in the link provided by you is not working now.
I tried using event[] syntax in ruby and it failed. I think the method to read & Store the events from outside ruby code has changed. Any idea how to access the events from the message ?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.