Check existence of a field and checking null value in field

Elastic Stack Logstash
1

I have different types of logs generated by the software we user in one big file. I am trying to assign "log_type" to different types of logs in my file. Since they have different number of columns(I used csv plugin to parse them), I tried to differentiate them by checking the existence of unique field. In the below example, the top event doesn't have column 16, but the bottom one does.

Screen Shot 2015-07-06 at 10.15.13 AM.png1200×782 182 KB

So my config looked like this:

if [column16] {
... "type1"
}else {
... "type2"
}

but as it shows in the picture, they got assigned the same type number, I think the above if statement doesn't work, it can't tell the difference between a null value in the field and the field doesn't exist.

any suggestions?

2

There does not seem to be a straightforward way to accomplish this.

I suggest using the ruby filter for now to call event.include?("[column16]") and add a tag based on that or something so that you can continue with your normal branch conditionals.

3

filed an issue for this here: https://github.com/elastic/logstash/issues/3568

4

Developing further the idea of using the ruby filter, can you try something like this?

                    code => "
					if event['column16'] == nil
						event['log_type'] = 'type2'
					else
						event['log_type'] = 'type1'
					end					
			"
	}```