CSV filter in logstash question

(John) #1

How would you add a tag to a column that has been parsed by the CSV plugin?

For instance, I have the following and it gets all the columns indexed, but it does not add tags to the matching [sourcetype] regex.

filter {
  if [type] == "csv" {
    csv {
      separator => ","
      columns => ["timestamp", "source", "sourcetype", "host", "index", "raw"]
      remove_field => ["message"]
      add_tag => ["csv"]
    grok {
      match => ["timestamp", "%{TIMESTAMP_ISO8601:timestamp} %{WORD:timezone}"]
      overwrite => ["timestamp"]
      #match => ["timestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
      match => ["timestamp", "ISO8601"]
    if [sourcetype] =~ /^vpc$/ {
      mutate {
        add_tag => ["vpc"]
    if [sourcetype] =~ /^iis$/ {
      mutate {
        add_tag => ["iis"]

If you are anchoring both the beginning and end of a string I think it would be clearer to use a simple string equality test.

if [sourcetype] == "vpc" {
(John) #3

The sourcetype column can have multiple entries for the string vpc, for instance it could be vpc1, vpc2... So I though I would have to regex match it?


Yes, but the ^ and $ anchor the regexp to start of string and end of string, so ^vpc$ only matches the exact string vpc.

If you want any string containing vpc use /vpc/. If you want any string beginning with vpc use /^vpc/.

(John) #5

Umm, yeah... Wow, it's been a long day. Thanks Badger for the sanity check!

(system) closed #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.