# Kibana 6 - Query/Highlight Performance

**URL:** https://discuss.elastic.co/t/kibana-6-query-highlight-performance/108038
**Category:** Kibana
**Created:** [November 16, 2017, 11:25pm UTC](https://discuss.elastic.co/t/kibana-6-query-highlight-performance/108038 "2017-11-16T23:25:24Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![fenneh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fenneh/32/24358_2.png) [@fenneh](https://discuss.elastic.co/u/fenneh)
#### Post date: [November 16, 2017, 11:25pm UTC](https://discuss.elastic.co/t/kibana-6-query-highlight-performance/108038/1 "2017-11-16T23:25:24Z")

</div>

Upgraded from 5.6.4 to 6.0.0 today to find query taking 5x the amount they have previously.

These queries are where there is no field specified, e.g. _term_

Digging around GitHub I couldn't find anything too similiar to what I'm experience reported.

I've found remedies in two different ways

- Disabling highlighting in Kibana
- Changing the default Kibana query string to use a specific field (e.g. message) as opposed to \*

This problem seems to exist on all our indexes (smallest has 400 fields, largest 1500) - They're not particularly big, \< 5GB in size.

5.6.4 does not exhibit the same behavior. I also do not use the \_all field so I simply can't update the default query to use that.

Related (I believe) Topics:

> [@Kibana very slow](https://discuss.elastic.co/t/kibana-very-slow/90157/7):
>
> If I query Level:\* instead of \* in Kibana its much quicker.

> <https://github.com/elastic/kibana/issues/12131>
>
> \*\*Kibana version\*\*:
> 6.0.0-alpha1
> 
> \*\*Elasticsearch version\*\*:
> 6.0.0-alpha1
> 
> …
> \*\*Server OS version\*\*:
> Ubuntu 16.04.2 LTS
> 
> \*\*Browser version\*\*:
> Safari 10.1.1 (12603.2.4)
> 
> \*\*Browser OS version\*\*:
> macOS 10.2.5
> 
> \*\*Original install method (e.g. download page, yum, from source, etc.)\*\*:
> download
> 
> \*\*Description of the problem including expected versus actual behavior\*\*:
> Discover view times out
> 
> \*\*Steps to reproduce\*\*:
> Open Discover view for a relatively small index (6.7K docs, 15.6MB data), but with a lot of fields (950 in my case) - the Discover view times out.
> 
> The data is from a pcap file, extracted using \`tshark -T ek\`.
> 
> \*\*Query executed (retrieved from slowlog)\*\*
> \`\`\`
> GET packets-2017-06-01/\_search
> {
> "size":500,
> "query":{
> "bool":{
> "must":\[
> {
> "query\_string": {
> "query":"\*",
> "fields":\[\],
> "use\_dis\_max":true,
> "tie\_breaker":0.0,
> "default\_operator":"or",
> "auto\_generate\_phrase\_queries":false,
> "max\_determinized\_states":10000,    
> "enable\_position\_increments":true,
> "fuzziness":"AUTO",
> "fuzzy\_prefix\_length":0,
> "fuzzy\_max\_expansions":50,
> "phrase\_slop":0,
> "analyze\_wildcard":true,
> "escape":false,
> "split\_on\_whitespace":true,
> "boost":1.0
> }
> },
> {
> "range": {
> "timestamp": {
> "from":null,
> "to":null,
> "include\_lower":true,
> "include\_upper":true,
> "boost":1.0
> }
> }
> }
> \],
> "adjust\_pure\_negative":true,
> "boost":1.0
> }
> },
> "version":true,
> "\_source":{
> "includes":\[\],
> "excludes":\[\]
> },
> "stored\_fields":"\*",
> "docvalue\_fields":\["timestamp"\],
> "script\_fields":{},
> "sort":\[
> {"timestamp":
> {"order":"desc",
> "unmapped\_type":"boolean"
> }}
> \],
> "aggregations":{
> "2":{
> "date\_histogram":{
> "field":"timestamp",
> "time\_zone":"America/Los\_Angeles",
> "interval":"135ms",
> "offset":0,
> "order":{
> "\_key":"asc"
> },
> "keyed":false,
> "min\_doc\_count":1
> }
> }
> },
> "highlight":{
> "pre\_tags":\["@kibana-highlighted-field@"\],
> "post\_tags":\["@/kibana-highlighted-field@"\],
> "fragment\_size":2147483647,
> "fields":{
> "\*":{
> "highlight\_query":{
> "bool":{
> "must":\[
> {
> "query\_string":{
> "query":"\*",
> "fields":\[\],
> "use\_dis\_max":true,
> "tie\_breaker":0.0,
> "default\_operator": "or",
> "auto\_generate\_phrase\_queries":false,
> "max\_determinized\_states":10000,
> "enable\_position\_increments":true,
> "fuzziness":"AUTO",
> "fuzzy\_prefix\_length":0,
> "fuzzy\_max\_expansions":50,
> "phrase\_slop":0,
> "analyze\_wildcard":true,
> "escape":false,
> "split\_on\_whitespace":true,
> "all\_fields":true,
> "boost":1.0
> }
> },
> {
> "range":{
> "timestamp":{
> "from":1331901000000,
> "to":1331901006792, 
> "include\_lower":true,
> "include\_upper":true,
> "format":"epoch\_millis",
> "boost":1.0
> }
> }
> }
> \],
> "adjust\_pure\_negative":true,
> "boost":1.0
> }
> }
> }
> }
> }
> }
> \`\`\`
> 
> \*\*Response when executed in Console (truncated)\*\*
> \`\`\`
> {
> "took": 100490,
> "timed\_out": false,
> "\_shards": {
> "total": 2,
> "successful": 2,
> "failed": 0
> },
> "hits": {
> "total": 6666,
> "max\_score": null,
> "hits": \[
> \`\`\`
> 
> \*\*Response from same query, without highlight\*\*
> \`\`\`
> {
> "took": 114,
> "timed\_out": false,
> "\_shards": {
> "total": 2,
> "successful": 2,
> "failed": 0
> },
> "hits": {
> "total": 6666,
> "max\_score": null,
> "hits": \[
> \`\`\`
> 
> 100 seconds vs. 114 milliseconds is quite the difference on a dataset that comfortably fits into memory.

> <https://github.com/elastic/kibana/issues/12097>
>
> \*\*Kibana version\*\*: 5.4.x
> 
> \*\*Elasticsearch version\*\*: 5.4.x
> 
> \*\*Server OS ver…sion\*\*: ANY
> 
> \*\*Browser version\*\*: ANY
> 
> \*\*Browser OS version\*\*: ANY
> 
> \*\*Original install method (e.g. download page, yum, from source, etc.)\*\*: ZIP file
> 
> \*\*Description of the problem including expected versus actual behavior\*\*:
> 
> Since https://github.com/elastic/elasticsearch/pull/20925 when executing a query string query in an index (index pattern) that had \`\_all\` disabled, we will look at all the fields in the mapping that are not metafields and can be searched, and automatically expand the list of fields that are going to be queried.
> 
> Basically, we will expand the query string for each specific field that you have in the index. Saved searches and discovery usually use the following query, along with other filters:
> 
> \`\`\`
> "query": {
> "query\_string": {
> "analyze\_wildcard": true,
> "query": "\*"
> }
> },
> \`\`\`
> 
> This means that for index patterns that have lots of fields, we will see the query expanded with many \`ConstantScore(\_field\_names:\<FIELD\_NAME\_HERE\>\` and this can be a performance impact compared with previous Elasticsearch/Kibana versions. This has been introduced in \`ES/Kibana 5.1.1\`.
> 
> This can be easily tested by grabbing the network call from a discovery page. This two steps are a quick repro:
> 
> \`\`\`
> PUT test
> {
> "mappings": {
> "test\_type": {
> "\_all": {
> "enabled": false
> },
> "properties": {
> "prop1": {
> "type": "text"
> },
> "prop2": {
> "type": "text"
> },
> "prop3": {
> "type": "text"
> },
> "prop4": {
> "type": "text"
> }
> }
> }
> }
> }
> 
> POST test/\_search
> {
> "profile": true,
> "version": true,
> "size": 500,
> "sort": \[
> {
> "\_score": {
> "order": "desc"
> }
> }
> \],
> "query": {
> "query\_string": {
> "analyze\_wildcard": true,
> "query": "\*"
> }
> },
> "\_source": {
> "excludes": \[\]
> },
> "stored\_fields": \[
> "\*"
> \],
> "script\_fields": {},
> "docvalue\_fields": \[\],
> "highlight": {
> "pre\_tags": \[
> "@kibana-highlighted-field@"
> \],
> "post\_tags": \[
> "@/kibana-highlighted-field@"
> \],
> "fields": {
> "\*": {
> "highlight\_query": {
> "query\_string": {
> "analyze\_wildcard": true,
> "query": "\*",
> "all\_fields": true
> }
> }
> }
> },
> "fragment\_size": 2147483647
> }
> }
> \`\`\`
> 
> Note that i added the \`profile: true\` to be able to see the lucene expressions being used. 
> 
> My proposal is to, when \`\_all\` is disabled and no \`fields\` or \`default\_field\` is used, add a warning that the query will auto expand to all the fields available in the index patter, and that will have an extra cost.

---

<div class="post-metadata">

### Author: ![Bargs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bargs/32/5429_2.png) [@Bargs](https://discuss.elastic.co/u/Bargs)
#### Post date: [November 17, 2017, 6:31pm UTC](https://discuss.elastic.co/t/kibana-6-query-highlight-performance/108038/2 "2017-11-17T18:31:47Z")

</div>

For the slow query, could you grab the raw query that's being sent from your browser's developer tools? You should see an `_msearch` request in the network tab.

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 18, 2017, 8:12am UTC](https://discuss.elastic.co/t/kibana-6-query-highlight-performance/108038/3 "2017-11-18T08:12:50Z")

</div>

I suspect this might be due to the fact that the `_all` field in version 6.0 has been deprecated and replaced with an [all\_fields option in the query string query](http://elastic-marketing). Instead of copying over data into a separate field that is indexed, which is what the `_all` field did, the new query iterates over all fields, meaning that data does not have to be indexed more than once but requiring more fields to be queried. It could be that you are seeing this as you have a reasonably large number of fields.

---

<div class="post-metadata">

### Author: ![fenneh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fenneh/32/24358_2.png) [@fenneh](https://discuss.elastic.co/u/fenneh)
#### Post date: [November 18, 2017, 8:46am UTC](https://discuss.elastic.co/t/kibana-6-query-highlight-performance/108038/4 "2017-11-18T08:46:50Z")

</div>

I did wonder that. But we've not been using \_all fields. The other odd thing I've noticed is that queries are now spanning over way more shards than previously.

E.g. if indexing every 24 hours (default logstash) I can query for the last 1 hour and the \_msearch returns saying it has queried all the shards for every logstash-\* index we have. Again, this isn't a behaviour I can see on 5.6.4.

I'll link the \_msearch response when I'm back in the office on Monday, the upgrade to 6.0 has had a few hiccups so far 😉

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 18, 2017, 8:54am UTC](https://discuss.elastic.co/t/kibana-6-query-highlight-performance/108038/5 "2017-11-18T08:54:52Z")

</div>

If you use query strings and do not specify a field, `_all` was used behind the scenes prior to 6.0. Querying more data and shards can naturally also affect performance. Make sure that you do not end top with a lot of small shards, as this can be inefficient.

---

<div class="post-metadata">

### Author: ![fenneh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fenneh/32/24358_2.png) [@fenneh](https://discuss.elastic.co/u/fenneh)
#### Post date: [November 18, 2017, 1:36pm UTC](https://discuss.elastic.co/t/kibana-6-query-highlight-performance/108038/6 "2017-11-18T13:36:52Z")

</div>

Oh really? That's interesting, I'll dig through one of the 5.x.x clusters we have running.

Regarding the shard thing - I'm seeing something which seems counter-intuitive to me. If a logstash index called logstash-2017.11.18 exists today and has 2 shards, and I query for the last 30 minutes data - I would expect to see in \_msearch a total of 2 shards queried.

What I am seeing instead is \_msearch return with a shard count which is the total number of shards for all logstash-\* indexes. How come the query is now checking previous indexes shards?

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 18, 2017, 1:51pm UTC](https://discuss.elastic.co/t/kibana-6-query-highlight-performance/108038/7 "2017-11-18T13:51:42Z")

</div>

In early 5.x versions, Kibana used the field stats API to identify exactly which indices to query. This replaced expanding date patterns.

In version 5.4, this API was deprecated as checking this at query time was made much more efficient, and these extra calls no longer required. I believe Kibana since then just queries against the index pattern, which might be why you see a larger number of shards respond than before.

---

<div class="post-metadata">

### Author: ![fenneh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fenneh/32/24358_2.png) [@fenneh](https://discuss.elastic.co/u/fenneh)
#### Post date: [November 18, 2017, 2:30pm UTC](https://discuss.elastic.co/t/kibana-6-query-highlight-performance/108038/8 "2017-11-18T14:30:04Z")

</div>

Thanks for the explanation on that, clarifies quite a few things. I'll be sure on Monday to post the \_msearch response/request.

As I said, for now, I've just set a default field for Kibana to query as opposed to it querying all fields.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 16, 2017, 2:30pm UTC](https://discuss.elastic.co/t/kibana-6-query-highlight-performance/108038/9 "2017-12-16T14:30:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

### Author: ![jordansissel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jordansissel/32/44957_2.png) [@jordansissel](https://discuss.elastic.co/u/jordansissel)
#### Post date: [April 24, 2018, 3:10am UTC](https://discuss.elastic.co/t/kibana-6-query-highlight-performance/108038/10 "2018-04-24T03:10:20Z")

</div>

I think I have experienced this as well. If I call `/_msearch` with the same query Kibana uses, it takes 5000ms. If I remove the `"highlight":...` part of the query, it returns in 100ms or less.

I tested on Elasticsearch 6.2.3, auditbeat 6.2.4 (provides the template).

Original query:

```auto
{"index":["infosec-auditbeat*"],"ignore_unavailable":true,"preference":1524525012740}
{"version":true,"size":500,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"_source":{"excludes":[]},"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"5m","time_zone":"America/Los_Angeles","min_doc_count":1}}},"stored_fields":["*"],"script_fields":{},"docvalue_fields":["@timestamp"],"query":{"bool":{"must":[{"query_string":{"query":"connect","analyze_wildcard":true,"default_field":"*"}},{"match_phrase":{"beat.hostname":{"query":"auditbeat-8mm7k"}}},{"range":{"@timestamp":{"gte":1524524346418,"lte":1524538746418,"format":"epoch_millis"}}}],"filter":[],"should":[],"must_not":[]}},"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"*":{}},"fragment_size":2147483647}}

```

`{"responses":[{"took":5267,...}`

And removing the highlight part:

```auto
{"index":["infosec-auditbeat*"],"ignore_unavailable":true,"preference":1524525012740}
{"version":true,"size":500,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"_source":{"excludes":[]},"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"5m","time_zone":"America/Los_Angeles","min_doc_count":1}}},"stored_fields":["*"],"script_fields":{},"docvalue_fields":["@timestamp"],"query":{"bool":{"must":[{"query_string":{"query":"connect","analyze_wildcard":true,"default_field":"*"}},{"match_phrase":{"beat.hostname":{"query":"auditbeat-8mm7k"}}},{"range":{"@timestamp":{"gte":1524524346418,"lte":1524538746418,"format":"epoch_millis"}}}],"filter":[],"should":[],"must_not":[]}}}

```

`{"responses":[{"took":25,...}`

As for my specific data:

```auto
GET /_cat/indices/infosec-auditbeat*

green open infosec-auditbeat-6.2.4-2018.04.24 7rGmI3A7T9anmVCphrflFw 5 1 713229 0 638.5mb 321mb
green open infosec-auditbeat-6.2.4-2018.04.23 YmY2OHc3RIOlkR1Xh1d0eA 5 1 329153 0 372.1mb 186.5mb

```

My mapping is moderate in size, `GET /infosec-auditbeat*/_mapping` (two indices) returns a JSON object which, when pretty-printed, is 2660 lines. This is the default auditbeat index template except for the index name changed.