Trigger output Exec once if file exists

Elastic Stack Logstash
1

I have this logstash conf. file:

input {
  file {
    path => "/usr/share/src-input/x_tran.csv"
    start_position => beginning
    sincedb_path => "/dev/null"
    mode => read
  }
}

output {
     exec {
         command => "code here"
         quiet => true
     }
}

in this case, the code in output will be triggered for every line in the csv file! Which it is not my case.

What I need to do is: once a file is detected -> trigger what is in the output once and once only!

is there any way to do that?

2

Use a multiline code with a pattern that never matches and a timeout.

3

sorry, i didn't get it! can you provide more details and where to use it, in filter or output?
@Badger
Would a multiline let the exec command run once per file?

4

Use a multiline codec on the input filter. Read this.

1 Like
5

I did this

input {
  file {
    path => "/usr/share/src-input/x_tran.csv"
    start_position => beginning
    sincedb_path => "/dev/null"
    codec => multiline {
       codec => multiline {
            pattern => "^Spalanzani" 
            what => "previous" 
            negate => true 
            auto_flush_interval => 5 
            max_lines => 200000000
    }
   }
  }
}
filter {

}
output {
    exec {
        command => "java -Xms256m -Xmx2048m -cp /var/script:/var/script/lib/* com.csvParser /usr/share/src-input/x_tran.csv /usr/share/src-output/output.csv"
    }
}

and it didn't trigger what's in the exec section at all! and it outputted the whole file to the log terminal several times!

Here's what's on the terminal @Badger :
The file I'm reading is quiet huge, 175 MB, and it outputs the whole content and then stops -> triggers the exec and then it re-loads the file again and so on!

109,-0.5782514288558133,0.050161287579731366,0.5445701020894942,-0.27990449217000146,-0.04159897836869265",
          "tags" => [
        [0] "multiline",
        [1] "multiline_codec_max_lines_reached"
    ],
    "@timestamp" => 2019-08-27T00:36:42.643Z,
          "path" => "/usr/share/src-input/x_tran.csv",
      "@version" => "1"
}
{
          "host" => "c630ebf27529",
       "message" => "74325,-0.82746815590408,0.6255582343669068,-0.5666050527906277,0.677479073905684,-0.4558699395151055,-1.2371385120679403,-0.40550146017102334,-0.92063 , .... ..... .... "
[0] "multiline",
        [1] "multiline_codec_max_bytes_reached"
}
[2019-08-27T00:13:50,199][INFO ][logstash.outputs.exec    ] 
[2019-08-27T00:13:50,400][INFO ][logstash.outputs.exec    ] 
[2019-08-27T00:13:50,625][INFO ][logstash.outputs.exec    ] 
[2019-08-27T00:13:50,831][INFO ][logstash.outputs.exec    ] 
[2019-08-27T00:13:51,036][INFO ][logstash.outputs.exec    ] 
[2019-08-27T00:13:51,239][INFO ][logstash.outputs.exec    ]

Note: the message field content is the content of the input file which it's 800k rows so I truncated it.

Here's how the first row of the csv look like:

row_number,Time_float,V1_float,V2_float,V3_float,V4_float,V5_float,V6_float,V7_float,V8_float,V9_float,V10_float,V11_float,V12_float,V13_float,V14_float,V15_float,V16_float,V17_float,V18_float,V19_float,V20_float,V21_float,V22_float,V23_float,V24_float,V25_float,V26_float,V27_float,V28_float,Amount_float,Class_float
0,-1.996583023457193,-0.6942423209592997,-0.04407492457044802,1.6727734992241514,0.973365514375461,-0.24511658354252203,0.34706794516190337,0.19367893831762997,0.0826372794108694,0.33112778320993075,0.08338554524255039,-0.540407035731003,-0.6182957177945313,-0.9960989219768981,-0.32461018632700356,1.6040138389062168,-0.5368328685192513,0.24486345402090126,0.030769932602201018,0.4962820266510568,0.3261180160164485,-0.024923364961491876,0.38285443833968436,-0.17691133433749112,0.1105069205607409,0.2465854429694212,-0.3921704315485548,0.3308916226487169,-0.06378115069750527,0.24496426337017338,-0.04159897836869265
6

You are hitting both the max lines limit and the max bytes limit so you are getting multiple events. And you can see that the exec output ran 6 times.

7

@Badger, the line number is less than the limit i have set in the codec filter.
All I want to do is just to be able to trigger one event per file, i don't want to read the file neither processing it. As you can see in the exec part I'm triggering a java code that read the file "hard-coded for now" and this part should be triggered once if the file exists only!
Any guidance please? I'm really lost!

8

Do not use logstash to do this. A shell script might be a better approach.

1 Like
9

Thanks a lot.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.