Logstash issue with dropping rows with a condition check on empty elements

[Nick11]

I have a CSV file with the following 2 rows (sample)

reservation date, reservationID
Jan 6th, res:id:adbcj-oksok-gjkk
Jan 10th,
Mar 10th, res:id:kkbcj-oksok-gjkk

My ask is to drop empty rows and apply a grok filter on reservationID to extract the last elements after the "-". This is what I did without success

csv {
        separator => ","
        skip_header => "true"
        autodetect_column_names => "true"
        skip_empty_columns => "true"
        skip_empty_rows => "true"
    }  

if [reservationID] =~ "" {
	grok {
		reservationID => "MY GROK PATTERN HERE, WHICH IS WORKING FINE EXTERNALLY THROUGH THE DEBUGGER"
	}
}

I was expecting the first and the third row in the output (not worried about the grok). Instead I see all 3 rows. Am I missing anything. I do not want the 2nd row in my output.

Thanks

[Badger]

You do not have any empty rows, so it will not skip any. The second line will not have a [reservationID] field (because you have set skip_empty_columns). So test that:

if ! [reservationID] { drop {} }

[Nick11]

Thanks for your reply.

I did try that and it did not work. I still get the 2nd row. I suspect that the skip_empty_columns condition is stripping the reservationID field even before I get a chance to do what you are suggesting.

[Badger]

That's the idea. That is meant to test whether the reservationID field exists, and dropping the event if it does not.

[Nick11]

Yes, makes sense. However, for the expression to evaluate the reservationID needs to be present.

What does skip_empty_colums exactly do? Does it strip out the column completely?

[Badger]

@Nick11

However, for the expression to evaluate the reservationID needs to be present

No, that is not correct.

If skip_empty_columns is set then columns containing no value will not get set.

[Nick11]

If that's the case then how can you use a conditional statement on that column in the next subsequent line.

Did you try your solution?

[Badger]

@Nick11

If that's the case then how can you use a conditional statement on that column in the next subsequent line.

The conditional is testing whether the field exists.

Note that your header row has a leading space on the column name so

if ! [reservationID] { drop {} }

will drop everything.

if ! [ reservationID] { drop {} }

will just drop the second row. And yes, I tested it.