Slow bulk api requests (ES 6.0 beta2)

[Jayme (Jayme Rotsaert)]

Hi,
I'm using 6.0 for testing.. but seeing that bulk update requests are very slow. :
A bulk update request with 300 items takes around 10 seconds average..
on a 800M docs index. Having a 8 node cluster with each server having 16, 2.4GHz processors, 40GB ram and SSDS so hardware is not the problem.

When looking at the hot_threads I see that a lot of time goes in to [warmer] calls..
I created my index with :

      "index" : {
        "refresh_interval" : "1s",
        "number_of_shards" : "16",
        "store" : {
          "preload" : [
            "*"
          ]
        },

So I suspect that this "store" "preload" is causing all the slowdowns ..

Is there a way to remove this setting from the index or do I need to reindex everything completely ? :S

Because I get :

{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: no settings to update;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: no settings to update;"},"status":400}J

Now when trying to run :

curl -XPUT 'elastic30:9200/my_index/_settings' -H 'Content-Type: application/json' -d'
{
  "index" : {
    "store.preload": []
  }
}'

Not 100% sure this is the cause of the slowdown but looking a the hot_threads I think it's a big suspect..
Thought "warmers" were completely removed from ES 5.X but apparently not.
Regards
Jayme

::: {elastic34}{g7ObCzVQRE2lMPH1wxI-2g}{1GjK_6CzRs2HkgdD8dcHew}{192.168.14.84}{192.168.14.84:9300}
   Hot threads at 2017-09-18T10:38:54.521Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   101.0% (504.9ms out of 500ms) cpu usage by thread 'elasticsearch[elastic34][warmer][T#5]'
     4/10 snapshots sharing following 21 elements
       org.apache.lucene.util.PriorityQueue.updateTop(PriorityQueue.java:211)
       org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:279)
       org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:301)
       org.apache.lucene.index.MultiDocValues$OrdinalMap.<init>(MultiDocValues.java:805)
       org.apache.lucene.index.MultiDocValues$OrdinalMap.build(MultiDocValues.java:762)
       org.apache.lucene.index.MultiDocValues$OrdinalMap.build(MultiDocValues.java:741)
       org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder.build(GlobalOrdinalsBuilder.java:65)
       org.elasticsearch.index.fielddata.plain.SortedSetDVOrdinalsIndexFieldData.localGlobalDirect(SortedSetDVOrdinalsIndexFieldData.java:127)
       org.elasticsearch.index.fielddata.plain.SortedSetDVOrdinalsIndexFieldData.localGlobalDirect(SortedSetDVOrdinalsIndexFieldData.java:45)
       org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.lambda$load$1(IndicesFieldDataCache.java:165)
       org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$$Lambda$1898/1260027281.load(Unknown Source)
       org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:401)
       org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:162)
       org.elasticsearch.index.fielddata.plain.SortedSetDVOrdinalsIndexFieldData.loadGlobal(SortedSetDVOrdinalsIndexFieldData.java:115)
       org.elasticsearch.index.fielddata.plain.SortedSetDVOrdinalsIndexFieldData.loadGlobal(SortedSetDVOrdinalsIndexFieldData.java:45)
       org.elasticsearch.index.IndexWarmer$FieldDataWarmer.lambda$warmReader$1(IndexWarmer.java:142)
       org.elasticsearch.index.IndexWarmer$FieldDataWarmer$$Lambda$1896/1627624627.run(Unknown Source)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:748)
     3/10 snapshots sharing following 18 elements
       org.apache.lucene.index.MultiDocValues$OrdinalMap.<init>(MultiDocValues.java:824)
       org.apache.lucene.index.MultiDocValues$OrdinalMap.build(MultiDocValues.java:762)
       org.apache.lucene.index.MultiDocValues$OrdinalMap.build(MultiDocValues.java:741)