Detecting empty fields not working as before in logstash 7.12.1

[mario_wood]

Hi,

I'm migrating our logstash from 6.8 to 7.12 and I'm finding that the detection of empty fields is not working as it used to.

For example given the input

{
  "foo":"",
  "bar":"baz"
}

and the logstash filter

if [foo] and [foo] == "" {
  mutate {
    remove_field => [ "foo" ]
  }
}

in logstash 6.8 we would have seen a document in elasticsearch like this:

{
  "bar":"baz"
}

but in logstash 7.12 it seems like it's not detecting the empty field so we see this in elasticsearch:

{
  "foo":"",
  "bar":"baz"
}

Are there some changes I've missed in the release notes that are affecting this, or should I be doing field detection another way?

We use this approach quite a lot and it's affecting our ability to detect and remove bad data from logs, or to coerce the data with the mutate filter to have default values.

Thanks,
Andy

[Badger]

Interesting. This will have changed in 7.0. With

input { generator { count => 1 lines => [ '' ] } }
filter {
    mutate { add_field => { "foo" => "" } }
    if [foo] { drop { } }
}
output  { stdout { codec => rubydebug { metadata => false } } }

by default the event is dropped. However, if you add --java_execution false then the event is not dropped.

[mario_wood]

Thanks, setting

...
pipeline.java_execution: false
...

In logstash.yml has restored the expected behaviour.

Would this be considered as a bug as there is a difference in behaviour when java_execution is used for creating an issue on github?

[Badger]

@mario_wood

Would this be considered as a bug

That is a matter of opinion. I think it is. I would try opening an issue on github and see if anyone agrees.