How do I conditionally parse a CSV into different columns?

Elastic Stack Logstash
1

I have CSV log files where the columns change from line to line. The first field is an eventCode, which I can use to know which subsequent columns to expect. Here's a representative example.

1,sameTaskId,username,size
2,sameTaskId,networkstats
1,anotherTaskId,username,size
3,sameTaskId,authAuditInfo
2,anotherTaskId,networkstats
3,anotherTaskId,authAuditInfo

(I also need to aggregate based on a TaskId, but that's another topic.)

How do I conditionally parse a line into the desired columns? I'm new to Logstash and to Ruby, so I'm stumbling a lot. With the below, I get an error Ruby exception occurred: undefined method '[]' for #<LogStash::Event:0x2bb54c8c> , which I guess is because I'm trying to get the first column value the wrong way.

ruby {
    code => "@metadata['eventCode'] = event['message'][0..event['message'].index(',')]"
}

if [@metadata][eventCode] == 1 {
    csv {
	    skip_header => true
        columns => ["myColA", "myColB"]
	}
} else if [@metadata][eventCode] == 2 {
    csv {
	    skip_header => true
        columns => ["myColA", "myColC"]
    }
}

Thanks!

2

I got something working using regular expressions! See below.

  if ([message] =~ /^1,/ ) {  
    csv {
	    skip_header => true
        columns => ["myColA", "myColB"]
	}
    }
  } else if ([message] =~ /^2,/ ) {   
       csv {
	    skip_header => true
        columns => ["myColA", "myColC"]
    }
  } else { drop {}}
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.