Exec output plugin

[Dor-Alter (Dor)]

I am trying to create a pipeline that takes a csv file as input write it out to a different file and then run some bash script on the output file and create a new output file.

Input_file.csv -> logstash -> temp_file -> ecex bash on temp_file -> logstash -> output_file

And the output file I will feed to elasticsearch.

Is this possible and if so how can I do that?

[leandrojmp (Leandro Pereira)]

@Dor-Alter

Is this possible and if so how can I do that?

I don't think this is possible, while you have an exec output plugin, it will output the same data that would be in the temp_file.

Also, not sure why you would need Logstash for that, you can output to a temp_file, have a bash script configured on your cron that would process that file and put in another directory where you would have another logstash pipeline reading it.

Depending on what your bash script will do, you may be able to do it directly on Logstash.

Please provide some more context of what you want to do with some sample data for your source and desired output.

[Dor-Alter (Dor)]

Ruby script with dynamic variables in logstash

Is it possible to have a dynamic variables in ruby that changes based on the input values I am getting from the inputfile? For example I have 5 input values which the attribute linked, if the value of link is 'connected' then I will need to increase the variable otherwise nothing. And for each value I will need to assign in a new attribute the current value of the variable. if the input is: username = test1, linked = disable, ... username = test2, linked = connected, ... username = test3, li…

This provides what I want to do with example.
I thought I can have the whole process with logstash without need to have some local exec running. As the data I am using will be modify and I want it to be up to date in elasticsearch