Hi,
I'm using 6.0 for testing.. but seeing that bulk update requests are very slow. :
A bulk update request with 300 items takes around 10 seconds average..
on a 800M docs index. Having a 8 node cluster with each server having 16, 2.4GHz processors, 40GB ram and SSDS so hardware is not the problem.
When looking at the hot_threads I see that a lot of time goes in to [warmer] calls..
I created my index with :
"index" : {
"refresh_interval" : "1s",
"number_of_shards" : "16",
"store" : {
"preload" : [
"*"
]
},
So I suspect that this "store" "preload" is causing all the slowdowns ..
Is there a way to remove this setting from the index or do I need to reindex everything completely ? :S
Because I get :
{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: no settings to update;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: no settings to update;"},"status":400}J
Now when trying to run :
curl -XPUT 'elastic30:9200/my_index/_settings' -H 'Content-Type: application/json' -d'
{
"index" : {
"store.preload": []
}
}'
Not 100% sure this is the cause of the slowdown but looking a the hot_threads I think it's a big suspect..
Thought "warmers" were completely removed from ES 5.X but apparently not.
Regards
Jayme
::: {elastic34}{g7ObCzVQRE2lMPH1wxI-2g}{1GjK_6CzRs2HkgdD8dcHew}{192.168.14.84}{192.168.14.84:9300}
Hot threads at 2017-09-18T10:38:54.521Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
101.0% (504.9ms out of 500ms) cpu usage by thread 'elasticsearch[elastic34][warmer][T#5]'
4/10 snapshots sharing following 21 elements
org.apache.lucene.util.PriorityQueue.updateTop(PriorityQueue.java:211)
org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:279)
org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:301)
org.apache.lucene.index.MultiDocValues$OrdinalMap.<init>(MultiDocValues.java:805)
org.apache.lucene.index.MultiDocValues$OrdinalMap.build(MultiDocValues.java:762)
org.apache.lucene.index.MultiDocValues$OrdinalMap.build(MultiDocValues.java:741)
org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder.build(GlobalOrdinalsBuilder.java:65)
org.elasticsearch.index.fielddata.plain.SortedSetDVOrdinalsIndexFieldData.localGlobalDirect(SortedSetDVOrdinalsIndexFieldData.java:127)
org.elasticsearch.index.fielddata.plain.SortedSetDVOrdinalsIndexFieldData.localGlobalDirect(SortedSetDVOrdinalsIndexFieldData.java:45)
org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.lambda$load$1(IndicesFieldDataCache.java:165)
org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$$Lambda$1898/1260027281.load(Unknown Source)
org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:401)
org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:162)
org.elasticsearch.index.fielddata.plain.SortedSetDVOrdinalsIndexFieldData.loadGlobal(SortedSetDVOrdinalsIndexFieldData.java:115)
org.elasticsearch.index.fielddata.plain.SortedSetDVOrdinalsIndexFieldData.loadGlobal(SortedSetDVOrdinalsIndexFieldData.java:45)
org.elasticsearch.index.IndexWarmer$FieldDataWarmer.lambda$warmReader$1(IndexWarmer.java:142)
org.elasticsearch.index.IndexWarmer$FieldDataWarmer$$Lambda$1896/1627624627.run(Unknown Source)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
3/10 snapshots sharing following 18 elements
org.apache.lucene.index.MultiDocValues$OrdinalMap.<init>(MultiDocValues.java:824)
org.apache.lucene.index.MultiDocValues$OrdinalMap.build(MultiDocValues.java:762)
org.apache.lucene.index.MultiDocValues$OrdinalMap.build(MultiDocValues.java:741)