High CPU load, many hot threads with the same stack

Elastic Stack Elasticsearch
1

I have an Elasticsearch cluster running very slow sometimes (>30s for each query) does to high CPU usage. In the hot threads, the thread I see the most is this one:

       sun.misc.Unsafe.park(Native Method)
       java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
       java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
       java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
       java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
       com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)
       com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
       com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137)
       com.google.common.cache.LocalCache$LoadingValueReference.waitForValue(LocalCache.java:3557)
       com.google.common.cache.LocalCache$Segment.waitForLoadingValue(LocalCache.java:2302)
       com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2191)
       com.google.common.cache.LocalCache.get(LocalCache.java:3937)
       com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
       org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:183)
       org.elasticsearch.index.fielddata.plain.SortedSetDVOrdinalsIndexFieldData.loadGlobal(SortedSetDVOrdinalsIndexFieldData.java:91)
       org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:144)
       org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals.globalMaxOrd(ValuesSource.java:117)
       org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:217)
       org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:64)
       org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:102)
       org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:87)
       org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:85)
       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)
       org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)
       org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)
       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
       org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
       org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:293)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)

How can I find the queries generating high CPU load, and how could I optimize it?

2

We can match the time this hot thread appears with the time a query is executed against ES. I find the discussion below to be very helpful.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.