Troubleshooting and improving slow Discover in Kibana 6

[A_B]

Hello all,

not 100% sure if this is a Kibana and/or Elasticsearch issue but I have to start somewhere...

Setup info:
We use Elastic Stack for centralized logging. I'm migrating from Elastic Stack 5.6.5 to Elastic Stack 6.3.2. At the moment I ship all our logs to both Elasticsearch clusters. Because Elasticsearch version 6 can only have one type of document per index I have sharded what used to be in 3 or 4 daily indices with 5 shards and one replica in Elasticsearch 5 to 12 daily indices with 2 shards and one replica. In the end the amount of shards is fairly similar per day.

The underlying hardware is very similar and the amount of ES and Kibana nodes is pretty much the same.

Issue:
Discover in Kibana 6.3.2 (searching ES 6.3.2) is very slow compared to Kibana 5.6.5 (searching ES 5.6.5).

In Kibana Monitoring I can see much higher Search Rate in Kibana 6 (30-50x compared to Kibana 5). Also search latency is much higher...

In Kibana 6 the Index pattern will match many more indices but around the same amount of shards.

Question:
Is the amount of indices a likely reason to the slow Kibana Discover in version 6.3.2? Any tips on what I can look at to find what is the root cause?

Cheers,
AB

[A_B]

Tested Kibana 6.1.4 without x-pack and it seems a bit quicker. Not like Kibana 5.6.5 with Elasticsearch 5.6.5 but better than Kibana 6.3.2...

[A_B]

I might also just be using it wrong...

Here is an example of a query that times out in Kibana 6.3.2 (takes ~ 5 sec in Kibana 5.6.5)
Query:
app: app1 AND ("some 4 letter string" OR "other 4 letter string" OR "3 letter string" OR "one more, string ")

Time 4h = 21M documents (of which 90 are matches)
Documents are typically 10-20 field log documents. Not particularly big documents. There are a fair amount of fields for the Index Pattern. (2,2K in Kibana 5.6.5, 1,3K in Kibana 6.3.2).

I would love to be able to decrease index count and field count but I do not have much control over that

[tsullivan (Tim Sullivan)]

In terms of Elasticsearch search performance, my understanding is that performance is affected more by the number of shards that a search needs to read from, and the size of the shards in GB (staying below 50GB per shard is anecdotally recommended).

Assuming worst-case scenario for everything:

Before:
  4 indices
   - 5 shards
   - 1 replica
  4 * (5 + 5) = 40 shards per day

After:
  12 indices
   - 2 shards
   - 1 replica
  12 * (2 + 2) = 48 shards per day

That doesn't seem like a huge difference.

One way to narrow down if the slowness is happening in Elasticsearch or in Kibana is reproduce the raw query in some other tool, preferably something minimal like curl. Kibana shows you the query it ran against Elasticsearch in its Spy Panel feature (replaced by Inspector in 6.4.x). The details from Elasticsearch show you the took time which is the internal time that Elasticsearch took to return the data, not counting network time. If that time is much shorter than the amount of time that Kibana took to render and parse the response, then Kibana is the culprit for being slow.

A large number of fields in the mappings are known to give Kibana problems, not limited to Discover. But that doesn't help explain why the performance would be different in 5.6.x vs 6.3.x - the parsing issues go back before version 5.

[Christian_Dahlqvist]

Given your query and the number of fields in your indices, I suspect this performance degradation might be related to the removal of the _all field. Up to and including Elasticsearch 5.x, all data was by default copied into an analysed system field called _all, and queries not specifying a field name would run against this. This often dramatically increased the index size of disk, but made it very fast to search for strings across all fields.

As shown in the blog post I linked to, this was deprecated in version 6.0 and replaced by the all_fields option in the query string query. This queries against all fields, which if you have a lot of them can have a significant impact on performance.

If you are willing to accept the larger index size on disk, you can create your own _all field using the copy_to functionality, and I think you then should be able to specify this new field as the default_field for query string queries, possibly through the advanced query:queryString:options setting in Kibana (I have not tried this).

[A_B]

Thank you very much for the suggestions @tsullivan and @Christian_Dahlqvist. I will run some tests and I might come back with more questions