Logstash issue with dropping rows with a condition check on empty elements

I have a CSV file with the following 2 rows (sample)

reservation date, reservationID
Jan 6th, res:id:adbcj-oksok-gjkk
Jan 10th,
Mar 10th, res:id:kkbcj-oksok-gjkk

My ask is to drop empty rows and apply a grok filter on reservationID to extract the last elements after the "-". This is what I did without success

csv {
        separator => ","
        skip_header => "true"
        autodetect_column_names => "true"
        skip_empty_columns => "true"
        skip_empty_rows => "true"

if [reservationID] =~ "" {
	grok {

I was expecting the first and the third row in the output (not worried about the grok). Instead I see all 3 rows. Am I missing anything. I do not want the 2nd row in my output.


You do not have any empty rows, so it will not skip any. The second line will not have a [reservationID] field (because you have set skip_empty_columns). So test that:

if ! [reservationID] { drop {} }

Thanks for your reply.

I did try that and it did not work. I still get the 2nd row. I suspect that the skip_empty_columns condition is stripping the reservationID field even before I get a chance to do what you are suggesting.

That's the idea. That is meant to test whether the reservationID field exists, and dropping the event if it does not.

Yes, makes sense. However, for the expression to evaluate the reservationID needs to be present.

What does skip_empty_colums exactly do? Does it strip out the column completely?

No, that is not correct.

If skip_empty_columns is set then columns containing no value will not get set.

If that's the case then how can you use a conditional statement on that column in the next subsequent line.

Did you try your solution?

The conditional is testing whether the field exists.

Note that your header row has a leading space on the column name so

if ! [reservationID] { drop {} }

will drop everything.

if ! [ reservationID] { drop {} }

will just drop the second row. And yes, I tested it.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.