CSV filter not working properly

Elastic Stack Logstash
1

Hi all, I have a folder which contains two different types of CSV files. My logstash config is as below. When I run logstash, it just ingests the csvs as is without the columns getting generated. If I explicitly mention the file, then it works. I reckon the problem is in the if condition as it doesn't match. Really appreciate if someone could point the issue here. Thank you!

input {
  file {
    path => "/opt/out/Vol/*.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
if [path] =~ "/opt/out/Vol/*pslist.PsList.csv"
{
  csv {
      separator => ","
      skip_header => "true"
      columns => ["TreeDepth","PID","PPID","ImageFileName","Offset(V)","Threads","Handles","SessionId","Wow64","CreateTime","ExitTime","File output"]
  }
  }
else if [path] =~ "/opt/out/Vol/*cmdline.CmdLine.csv"
{
  csv {
      separator => ","
      skip_header => "true"
      columns => ["TreeDepth","PID","Process","Args"]
  }
}

}
output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "vol"
  }

stdout {}

}
2

I'd try something like this.

  if "pslist" in [path] {
    # CSV Filter
  }
  else if "cmdline" in [path] {
    # CSV Filter
  }

or if it's just 2 different CSV types then make 2 different inputs and then tag them so you know which is which. Then you can use type for conditional statements.

input {
  file {
    path => "/opt/out/Vol/path-to-get-first-type"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    type => "first"
  }
  file {
    path => "/opt/out/Vol/path-to-get-second-type"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    type => "second"
  }
}
filter {
  if [type] == "first" {
    # CSV Filter
  }
  else if [type] == "second" {
    # CSV Filter
  }   
}
3

@aaron-nimocks - Thanks a lot. The first solution worked like a charm! I was trying all sorts of things to fix this but none worked.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.