ES Server high CPU on CloseableThreadLocal

Elastic Stack Elasticsearch
1

Hi,

I have 2 servers out of 3 that have high constant CPU usage due to this hot thread:

83.8% (419.2ms out of 500ms) cpu usage by thread 'elasticsearch[elasticsearch-003][flush][T#5]'
     6/10 snapshots sharing following 10 elements
       org.apache.lucene.util.CloseableThreadLocal.set(CloseableThreadLocal.java:97)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextThreadLocal.set(ThreadContext.java:509)
       org.elasticsearch.common.util.concurrent.ThreadContext.lambda$stashContext$0(ThreadContext.java:109)
       org.elasticsearch.common.util.concurrent.ThreadContext$$Lambda$1477/1354584016.close(Unknown Source)
       org.elasticsearch.common.util.concurrent.ThreadContext$StoredContext.restore(ThreadContext.java:348)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onAfter(ThreadContext.java:616)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:41)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 10 elements
       org.apache.lucene.util.CloseableThreadLocal.set(CloseableThreadLocal.java:97)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextThreadLocal.set(ThreadContext.java:511)
       org.elasticsearch.common.util.concurrent.ThreadContext.lambda$newStoredContext$2(ThreadContext.java:135)
       org.elasticsearch.common.util.concurrent.ThreadContext$$Lambda$1476/657820185.close(Unknown Source)
       org.elasticsearch.common.util.concurrent.ThreadContext$StoredContext.restore(ThreadContext.java:348)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:636)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 8 elements
       org.apache.lucene.util.CloseableThreadLocal.set(CloseableThreadLocal.java:97)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextThreadLocal.set(ThreadContext.java:511)
       org.elasticsearch.common.util.concurrent.ThreadContext.stashContext(ThreadContext.java:108)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:634)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)

Which i think is caused by a bug.
Btw I'm using the azure-plugin.
ES 6.0.0-GA

2

Can you tell us more about your workload? Are you heavily indexing, searching or reindexing on this cluster maybe?

3

It was after an upgrade to 6.0.0GA from 6.0.0-rc1.

Those specific servers don't have any real time indexing to them. Every 30 minutes they get a bulk update of some docs and thats it.

Searching is really low, a few requests a minute

4

Btw, restarting the machines fixed it

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.