Elasticsearch hot threads Interpretation

Hi,
I I am having ES(version 1.7.5) cluster of 6 nodes, 3 is kept for read and rest for write. I have a scenario where the usage of parent child concept with bulk indexing is required.At irregular intervals CPU usage of majority of nodes go above 90%. I couldn't find any direct relation to any of my search or indexing activity. When i ran hot threads API i get following results.

101.4% (506.7ms out of 500ms) cpu usage by thread 'elasticsearch[node1][search][T#4]'
2/10 snapshots sharing following 47 elements
org.apache.lucene.util.packed.Packed64.set(Packed64.java:223)
org.apache.lucene.util.packed.PackedLongValues$Builder.pack(PackedLongValues.java:268)
org.apache.lucene.util.packed.DeltaPackedLongValues$Builder.pack(DeltaPackedLongValues.java:89)
org.apache.lucene.util.packed.MonotonicLongValues$Builder.pack(MonotonicLongValues.java:91)
org.apache.lucene.util.packed.PackedLongValues$Builder.pack(PackedLongValues.java:244)
org.apache.lucene.util.packed.PackedLongValues$Builder.add(PackedLongValues.java:227)
org.apache.lucene.index.MultiDocValues$OrdinalMap.(MultiDocValues.java:550)
org.apache.lucene.index.MultiDocValues$OrdinalMap.build(MultiDocValues.java:482)
org.apache.lucene.index.MultiDocValues$OrdinalMap.build(MultiDocValues.java:446)
org.elasticsearch.index.fielddata.plain.ParentChildIndexFieldData.buildOrdinalMap(ParentChildIndexFieldData.java:282)
org.elasticsearch.index.fielddata.plain.ParentChildIndexFieldData.localGlobalDirect(ParentChildIndexFieldData.java:312)
org.elasticsearch.index.fielddata.plain.ParentChildIndexFieldData.localGlobalDirect(ParentChildIndexFieldData.java:65)
org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$2.call(IndicesFieldDataCache.java:211)
org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$2.call(IndicesFieldDataCache.java:199)
org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:199)
org.elasticsearch.index.fielddata.plain.ParentChildIndexFieldData.loadGlobal(ParentChildIndexFieldData.java:267)
org.elasticsearch.index.search.child.ChildrenConstantScoreQuery.createWeight(ChildrenConstantScoreQuery.java:96)
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:684)
org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:133)
org.elasticsearch.index.search.child.CustomQueryWrappingFilter.getDocIdSet(CustomQueryWrappingFilter.java:72)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:157)
org.apache.lucene.search.Weight.bulkScorer(Weight.java:131)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.bulkScorer(ConstantScoreQuery.java:141)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:301)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:312)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)

Hey,

this looks, as if during a search the data structure for parent/child is being recreated. You can change this to ensure it is loaded eagerly using global ordinals for example (note, be careful with this option if you have high indexing rates, take your time to play around with it).

See https://www.elastic.co/guide/en/elasticsearch/reference/1.7/fielddata-formats.html#_global_ordinals

--Alex

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.