Logstash pipe plugin exec not working

(Sharat Jagannath) #1

I have a logstash conf file that gets data from kafka as input and then based on if the input contains a field it is supposed to pipe the input data to a script(unix of python)
For example-

kafka {
zk_connect => ''
     topic_id => 'READY_FOR_INDEX'
 kafka {
      zk_connect => ''
      topic_id => 'INDEX_CSV'
  output {
if ([csvLocation]) {
          command => "/home/test.sh"
stdout {codec => rubydebug}


The script reads the input as arguments i suppose

 while read LINE; do
   echo ${LINE}    # do something with it here

 exit 0

But for some reason the script does not get any data. How do i get this working to get all the input data.
I was referring to https://www.elastic.co/guide/en/logstash/current/plugins-outputs-pipe.html


Pipe output plugin not passing event
(Magnus Bäck) #2

Please show an example event (i.e. output from your stdout {codec => rubydebug} output). Is Logstash forking a child process that runs /home/test.sh? Is there anything in Logstash's log? You might want to increase the log level to see everything that's interesting.

(Sharat Jagannath) #3

@magnusbaeck Here us a sample stdout event.

              "index" => "xxx",
     "dimension_name" => "xxx",
             "source" => "xxx",
               "type" => "xxx",
"abc" => {
                        "test1" => {
          "display_name" => "xxx",
        "testid" => "5xxx"
                            "test2" => {
          "display_name" => "xxx2",
        "testid" => "6xxx"
         "PROJECT_ID" => "xxx",
          "tableName" => "xxx",
        "csvLocation" => "abc.csv",
           "@version" => "1",
         "@timestamp" => "2017-01-19T21:24:21.775Z"

this is what i get when i run logstash in debug mode

Opening pipe {:command=>"/home/test.sh", :level=>:info}
Starting stale pipes cleanup cycle {:pipes=>{"python /home/test.sh"=>#<PipeWrapper:0x7150cc3 @pipe=#<IO:fd 454>, @active=true>}, :level=>:info}

(Magnus Bäck) #4

But that's not the only output, right? Off the top of my head I don't know what's up here.

(Sharat Jagannath) #5

Sorry I didn't get you. I didn't post the actual output for privacy reasons.but its mostly structured that way.

(Magnus Bäck) #6

Well, seeing more of the logs could be helpful. You can obfuscate sensitive parts if you like

(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.