Problem updating settings: maxThreadCount (= 8) should be <= maxMergeCount (= 6)

Elastic Stack Elasticsearch
1

Hi there,

(let me know if I file a bug on Github directly)

We're in the process of upgrading to ES5 (5.0.2 in our case). We have reindexed a lot of data already.

I wanted to try settings I saw in a blog post from a friend (adapted), so I issued this command:

$ curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
  "index.merge.scheduler.max_thread_count" : "8",
  "index.merge.policy.max_merge_at_once" : "4",
  "index.merge.policy.max_merge_at_once_explicit" : "4",
  "index.merge.policy.max_merged_segment" : "20gb",
  "index.merge.policy.segments_per_tier" : "4"
}'

The command worked fine. After that I couldn't update any setting anymore:

$ curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
 "index.merge.scheduler.max_thread_count" : "17"
}'
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[hostname.example.net][10.1.4.188:9300][indices:admin/settings/update]"}],"type":"illegal_argument_exception","reason":"maxThreadCount (= 8) should be <= maxMergeCount (= 6)"},"status":400}

I tried with "index.merge.scheduler.max_merge_count" too, and any other param. I tried every possible value below, equal, or above 6/8. I tried other params.

Despaired, I shutdown the whole cluster, and restarted nodes. My indices won't open anymore. I saw the params are kept in ".st" files in master and data nodes, but I don't know how they can be edited easily.

My current understanding of the problem is:

  • ES accepted params it shouldn't have accepted in the first place
  • current settings prevent any setting modifications
  • if I read the code correctly, "MergeSchedulerConfig::setMaxThreadAndMergeCount()" is called on instantiation with the current params, so I don't have any hope of recovering my index (with 180M docs in it) without going back to a valid "state" file.
  • my other option is to recompile ES without this safety check just to open my indices and configure this setting to a right value and go back to an official .jar afterwards

What do you think? I find this issue on GH: https://github.com/elastic/elasticsearch/issues/20380 / https://github.com/elastic/elasticsearch/pull/20383, but it's supposed to be in 5.0.0, and my problem happens on 5.0.2.

Thanks,

Jean-Baptiste

2

Thanks for reporting Jean-Baptiste.
I opened https://github.com/elastic/elasticsearch/pull/23391 for the fix.

 my other option is to recompile ES without this safety check just to open my indices and configure this setting to a right value and go back to an official .jar afterwards

Yes this is the only option I see at the moment, sorry for the trouble. The fix will be in 5.3 hopefully.

3

@jimczi thanks for your answer and the PR! Happy to help pushing it if needed, let me know what I can do to help.

I ended up rebuilding a jar with the a few guard lines commented out and I could change the setting to a safe value and get my data back. It was surprisingly easy btw, I'll write a quick article on that, it will hopefully help other people needing that kind of sorcery. Thanks!