CSV filter in logstash question


(John) #1

How would you add a tag to a column that has been parsed by the CSV plugin?

For instance, I have the following and it gets all the columns indexed, but it does not add tags to the matching [sourcetype] regex.

filter {
  if [type] == "csv" {
    csv {
      separator => ","
      columns => ["timestamp", "source", "sourcetype", "host", "index", "raw"]
      remove_field => ["message"]
      add_tag => ["csv"]
    }
    grok {
      match => ["timestamp", "%{TIMESTAMP_ISO8601:timestamp} %{WORD:timezone}"]
      overwrite => ["timestamp"]
    }
    date{
      #match => ["timestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
      match => ["timestamp", "ISO8601"]
    }
    if [sourcetype] =~ /^vpc$/ {
      mutate {
        add_tag => ["vpc"]
      }
    }
    if [sourcetype] =~ /^iis$/ {
      mutate {
        add_tag => ["iis"]
      }
    }
  }
}

#2

If you are anchoring both the beginning and end of a string I think it would be clearer to use a simple string equality test.

if [sourcetype] == "vpc" {
    [...]

(John) #3

The sourcetype column can have multiple entries for the string vpc, for instance it could be vpc1, vpc2... So I though I would have to regex match it?


#4

Yes, but the ^ and $ anchor the regexp to start of string and end of string, so ^vpc$ only matches the exact string vpc.

If you want any string containing vpc use /vpc/. If you want any string beginning with vpc use /^vpc/.


(John) #5

Umm, yeah... Wow, it's been a long day. Thanks Badger for the sanity check!