Logstash csv - Filters only rows with specific column count

[Lubos_Marek (Luboš Marek)]

Hello,
I have a log file (*.csv) with this specifific structure
datetime;level; statuscode; message

Sometimes missing the "statuscode" and that row has only 3 columns (datetime;level;message).

Can I somehow skip rows with 3 columns?

input {

  file {

    path => ["/usr/share/logstash/data1/*.csv"]

    start_position => "beginning"

    sincedb_path => "/dev/null"

  }

}

filter {

  csv {

    separator => ";"

    columns => ["datetime", "level", "statuscode", "message"]

  }

}

output {

  elasticsearch {

    hosts => ["http://host.docker.internal:9200"]

    index => "log"

  }

}

[rugenl (Len Rugen)]

Something like this filter might work:

  ruby {
    code => 'if event.get("[message]).count(\";\") < 3
      event.cancel
    end'
  }

"Might" because some of the examples I used were the "old" ruby syntax before we needed to use "event.get".

The idea is that if there are less than 3 separator chars, there aren't 4 columns.

That CSV should have ";;" to designate the statuscode is null instead of this format.