Disabling TieredCompilation leads to timeout

Elastic Stack Elasticsearch
1

Hello,

I actually hesitate to create an issue on github but I think I found an issue which can impact several deployments.

TLDR : Disabling TieredCompilation on Elasticsearch 5.5 to 6.1 can lead to:

  • Increased time for stats query (correlated with the number of shards or the number of indices)
  • Timeouts on DateHistogramAggregation when time_zone is enabled
  • Timeouts with xpack monitoring
  • After sometime (between 10h and 1 day)

Two articles from Elastic more or less explain than disabling TieredCompilation is not detrimental:
https://www.elastic.co/blog/memory-issues-well-remember, https://www.elastic.co/blog/honey-you-have-changed-quite-a-bit

Recently, I setup/upgraded elasticsearch clusters (6.1, 5.6) with TieredCompilation disabled (XX:-TieredCompilation in jvm.options) and noticed the following symptoms:

  • A stats query could last multiple seconds, even almost 2 minutes
  • I had a lot of timeouts with marvel/xpack monitoring
  • DateHistogramAggregation were awfully long and lead to timeout when time_zone was used (default in Kibana) - it worked without time_zone
  • All these issues disappeared for ~1 day after a cluster restart and reappeared after (I suspect it's related to index creation but not sure)

My first suspect was to say it was a 6.X issue. When I got the same thing with the upgrade of another cluster from 5.5.2 to 5.6.8, I thought it was a shard issue (I had 8000 shards on my 5.6 cluster, wheras the rule of < 25*GB_HEAP should have restrain me at ~2000). When I decreased the number of shards to 1400, and them 1000, and the issue was still there, I thought it was something which appeared in 5.6).

But the stats query time decreased from 1m40s to 10s with the decrease of shards (but was still too long)

When I setup a new cluster in 5.5 and still got the issue (and not on a cluster in 5.2 setup 1 year ago), I noticed the main difference was the disabling of TieredCompilation and enabling it back (removing XX:-TieredCompilation) fixed all the issues, in 5.5 and 6.1 (I don't have anymore the 5.6 cluster, I reindexed everything in a new 5.5.2).

I wonder if the TieredCompilation disabling would be related to the issues with Joda:

Regards,
Grégoire

2

Bump, we are also interested in this.

The last time I experimented with disabling TieredCompilation within our 26-node cluster I also had catastrophic results with x-pack and timeouts.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.