Option for excluding fields in "all_fields" mode query for ES 6.0

Elastic Stack Elasticsearch
1

In ES 6.0 the indexed "_all" field will be removed and Elastic is promoting the "all_fields" query that does the same (at query- and not indexed time).

Works fine for us, but there is one part missing:
For the "_all" field you could exclude fields by specifying "include_in_all": false. The same configuration needs to be available for the "all_fields" searches. Otherwise it is not possible to migrate the _all-index-field feature to the new "all_fields"-query.

There were many other complains about this. E.g. Exclude particular fields from [all_field] searches

Is there a solution for this? Are your ES-guys working on this? Or did I miss something?

Thanks
Mos

2

There isn't a replacement for include_in_all: false with the all_fields mode. Instead, you have two options - you can manually specify which fields to use for querying by specifying the fields (ie, not using all_fields mode), or you can use the copy_to mapping parameter to copy only the fields you need to a new field (say called "my_all_field") and then set the default_field to my_all_field in either the index settings or query parameters.

3

Thanks @dakrone for helping clarifying this.

In our case (which I think is common), we have the following scenario:
We have around 40 fields from which are 35 copied into the _all field. So we use the include_in_all: false for 5 fields in our index.
We were looking forward to the the "all_fields" query because it has some advantages over the _all field as you presented on your Elastic{ON} talk (smaller index, usage of analyzer ....).
But to use it, we need some kind of excluding those fields. (Sure, otherwise we need to stay with the copy_to function).

Why didn't you decide to implement a replacement for "include_in_all: false"? (Would be consistent if the recommendation is to use the all_fields query instead of the _all field)

If we use your suggestion "manually specify which fields to use for querying" are we able to simulate exactly what the "all_fields" query does? How would it look like?

4

Why didn't you decide to implement a replacement for "include_in_all: false"? (Would be consistent if the recommendation is to use the all_fields query instead of the _all field)

It wasn't for any reason, it was simply to start with the simplest implementation, since we expect it to change as we go (and it is changing, see: Refactor field expansion for match, multi_match and query_string query by jimczi · Pull Request #25726 · elastic/elasticsearch · GitHub)

If we use your suggestion "manually specify which fields to use for querying" are we able to simulate exactly what the "all_fields" query does? How would it look like?

It would mean that you do:

POST /_search
{
  "query": {
    "query_string": {
      "query": "foo bar baz",
      "fields": ["a*", "b", "cd*"],
      "lenient": true
    }
  }
}

(of course you will need to fill in the fields with the real fields you want to query.

5

Ok, thanks. The query wouldn't work for us. It would assume that the ES client knows about all fields that should be searched by default. That's not the case.
As long as a "include_in_all:false" is not supported for the all_fields mode it seems not to be a realistic replacement for the _all field.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.