Search over all fields with all_fields and default_field differ

Elastic Stack Kibana
1

Hi,

we are using Kibana 5.4.1 (and so ES) and we need to search over all fields without the _all field in Kibana Discover.

As solution we thought that we could set the Kibana option: "query:queryString:options" to
{ "analyze_wildcard": true,
"all_fields" : true,
"lenient": true,
"default_operator": "and" }

This search configuration works well (e.g. search for a simple string consisting of multiple words like My Company 1) and has the correct search results. Unfortunately, the "all_fields" is deprecated as of 6.0.0, and we want to be future-proof.
https://www.elastic.co/guide/en/elasticsearch/reference/6.x/query-dsl-query-string-query.html

Therefore, we used instead of "all_fields" the property "default_field", as suggested by the documentation above, like this:

{ "analyze_wildcard": true,
"default_field": "*",
"lenient": true, 
"default_operator": "and" }

The query (excerpt) that Kibana is sending (al_fields is automatically added and set to false):

{ "query_string": {
"all_fields":false,
"analyze_wildcard":true,
"default_field":"*",
"default_operator":"and",
"lenient":true,
"query":"My Company 1"}
}

Unfortunately, the search result is completely different. It finds nearly every entity in our database (like searching for everything) instead of a sub set.

So, is there a solution to this search so we are compatibile with the 6.0 version? Or do we need to change from "all_fields" to "default_field" only if using Kibana/ES 6.0 ?

Thanks
Christian Wimmer

2

Hi @WimmerELO

You shouldn't need to set all_fields manually in the query:queryString:options setting. If you're querying against an index without an _all field, all_fields mode will be used automatically (unless you specify a default_field either in the index settings or the query). Is there some reason this didn't work for you?

The same will be true once all_fields is removed. default_field: * will become the default and will work the same way all_fields: true did. Again, this shouldn't require any changes on your part, it should work automatically.

If you were testing default_field: * in 5.x it likely returned errors or poor results. This was only fixed in 6.0: https://github.com/elastic/elasticsearch/pull/25726.

Note that all_fields was only deprecated in 6.0, it won't be removed until 7.0.

3

Thanks for your answer !

After some investigation:
We cannot leave out all_fields : true because existing ES indexes have empty _all field in its mapping already. Due to big data it is not convenient to reindex only for this.

default_field and fields with value of "*" return to many results because a lot more fields are checked than when using all_fields, it seems (the explain syntax shows this).

When version 7 is near release, we will check for compatibility.

Thanks
Christian Wimmer

4

Do you actually need all_fields for the indices that have an _all field? If you don't specify all_fields in your query Elasticsearch will automatically use _all for indices where it exists and all_fields where it does not.

Nothing wrong with setting all_fields if that's what you want though. Please do try default_field: * in 6.0 and let us know if you run into problems.

5

We do not want to use _all fields at all, but due to not knowing beforehand, we accidentally added the _all field earlier, and now the field is always included in our data but empty. We could reindex, but this takes a lot of time for some customers.

We use all_fields for now and keep the issue in mind for later releases.

Thanks!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.