Why is catch-all field slower in 6.2 compared to 2.4?

[ar21 (Arun)]

We upgraded a 2.4 cluster to 6.2 cluster using the reindex from remote approach. In 2.4, we were using the catch-all _all field to perform searches and were seeing response times under 500 ms for all our queries.

In 6.2, the _all field is no longer available for the new index, so we ended up creating a new text type field called all like "all": {"type": "text"} and set copy_to on all our other fields (about 2000 of them). But now, searches on this new catch-all field all are taking 2 to 10 times longer than the search on the 2.4 _all field. (We flushed the caches on both clusters before performing the queries.)

Both clusters are single data center, single node 8GB memory on the same AWS zone, hosted through elastic cloud. Both indices have the same number of documents (about 6M) and have about 150 Lucene segment files.

The 2.4 cluster though is being used by a staging app, which might send some queries to it every few minutes, when testers are testing things. Is there any caching (other than ES caches themselves) that might be the reason for this? I am still at a loss as to why the 6.2 queries are this much slower.

[diranged]

@Ar21, did you ever resolve this? We also are migrating from 2.4 -> 6.2 and seeing slow catch all queries. We're not using the _all field directly, but we see in Kibana 6.2 that if we search for something simple like "cron:session" the result will take 50+ seconds to come back. If we are specific and search for some_message_field:"cron:session" then the result comes back in ~800ms.

[ar21 (Arun)]

No, I didn't get an answer from anyone at elastic so far (as you can see here). I didn't have a chance to benchmark it more thoroughly either, so I don't have an answer.

In your case ensure that Elasticsearch isn't thinking that cron is a field name since it is immediately followed by a :.

[diranged]

So I actually figured it out on our end. When you don't specify a field name, in ES 6 and Kibana 6, a default search parameter default_field: "*" is added by Kibana. Even if Kibana did not add that, ElasticSearch would add default_field: "*.*". (**)

To resolve this, we added a copy_to: "all" mapping to all of our fields, and then reconfigured Kibana to useallas thedefault_field`. This resolves the speed issues entirely.

Unfortunately, we don;'t have the mapping quite right yet.. so we are still having issues. I opened up another post about it here: Proper Template Mapping for ES/Kibana 6.2.3

**: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html