ELK 6.7.0: Search query match on inexisting fields

[obuez]

Hi all,

I just upgrade elasticsearch and kibana from version 6.5.0 to version 6.7.0
There is an impact on my application as a specific query in kibana is not interpreted the same way as before.

ex: field1: "test" AND field2: "test"
Would return every documents containing field1 and field2 with the specified values "test". If a document only contains field1 or field2 (with value "test") with the other absent, it wouldn't be returned.

With kibana version 6.7.0, if a document only contains field1 or field2 (with the correct value "test") and the other absent, it would be returned.
ex:

{
  "query": {
    "bool": {
      "filter": [
        {
          "query_string": {
            "query": "field1:test AND field2:test"
          }
        }
      ]
    }
  }
}

From version 6.7.0 will also return events like:

{
    "field1": "test"
},
{
    "field1": "test",
    "field2": "test"
},
{
    "field2": "test"
}

Whereas before (6.5.0) it would only have returned:

{
     "field1": "test",
     "field2": "test"
}

Which is definitely screwing-up our expectations and application.
I'm just wondering if this is a bug (I need to downgrade, or wait for a fix) or a feature (I need to refactor my application). Or ... ?

Thanks for your feedback

[Bill_McConaghy]

I'm moving this to the Elasticsearch forum as I can reproduce this behavior in Console in Dev Tools going directly against Elasticsearch.

[daanraman]

For anyone running into the same issue: we had to hot-patch all the queries in our product with an "exists:" clause to return to the old behavior.

For example, instead of NonExistingField1:test AND NonExistingField2:test now becomes: _exists_:NonExistingField1 AND _exists_:NonExistingField2 AND NonExistingField1:test AND NonExistingField2:test which returns 0 events as you would expect.

[Bill_McConaghy]

Thanks for the workaround suggestion @daanraman !