Hi Elastic community,
I have a cloud cluster with 3 hot nodes. Stack monitoring data is showing me that 2 out of 3 are constantly under heavy load. CPU usage is nearly 100% all the time. I would like to know how can I track what activity is causing this and how can I balance the load more evenly accross my cluster before thinking about increasing the resources.
Thank you,
Philippe
The nodes hot threads API might be a good start. See Nodes hot threads API | Elasticsearch Guide [8.1] | Elastic
Thank you for your reply
This API produces a big ouput. Since I am really new to Elastic, I don't know where to start an more importantly, don't know how to interpret the results. Is there any documentation out there?
Thank you again!
Philippe
Feel feel to put the data into a pastebin, happy to take a quick look.
It would also help if you tell a little more about your setup (Elasticsearch version, which operating system, hardwrare being used, specific setup) and what kind of load you are putting against the cluster.
My organization and I have a 3 nodes Elastic Cloud cluster version 8.1 with 15gb or RAM and 450gb of disk. Our use case is to use Elasticsearch as a SIEM. We are currently in the phase of collecting all of our on premise instances. We have 154 Elastic agents to gather and send data to the cluster. Data retention is configured to be 30 days then it is deleted. We will need to keep data for a longer period of time in a near future.
Now here's the result of the node hot threads API for one of the busiest node:
Thanks!
100.0% [cpu=65.6%, other=34.4%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[instance-0000000001][write][T#3]'
7/10 snapshots sharing following 23 elements
app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:239)
app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1530)
app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1815)
app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1468)
app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1256)
app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1192)
app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1040)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:973)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:891)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:320)
app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:250)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base@17.0.2/java.lang.Thread.run(Thread.java:833)
2/10 snapshots sharing following 30 elements
app//org.elasticsearch.index.mapper.ContentPath.pathAsText(ContentPath.java:59)
app//org.elasticsearch.index.mapper.DocumentParser.getMapper(DocumentParser.java:606)
app//org.elasticsearch.index.mapper.DocumentParser.getLeafMapper(DocumentParser.java:619)
app//org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:506)
app//org.elasticsearch.index.mapper.DocumentParser.parseNonDynamicArray(DocumentParser.java:485)
app//org.elasticsearch.index.mapper.DocumentParser.parseArray(DocumentParser.java:428)
app//org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:284)
app//org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:260)
app//org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:354)
app//org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:387)
app//org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:282)
app//org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:260)
app//org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:118)
app//org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:80)
app//org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:81)
app//org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:998)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:947)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:891)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:320)
app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:250)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base@17.0.2/java.lang.Thread.run(Thread.java:833)
unique snapshot
app//com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:294)
app//org.elasticsearch.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:76)
app//org.elasticsearch.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:249)
app//org.elasticsearch.xcontent.DelegatingXContentParser.textOrNull(DelegatingXContentParser.java:92)
app//org.elasticsearch.index.mapper.KeywordFieldMapper.parseCreateField(KeywordFieldMapper.java:765)
app//org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:228)
app//org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:356)
app//org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:508)
app//org.elasticsearch.index.mapper.DocumentParser.parseNonDynamicArray(DocumentParser.java:485)
app//org.elasticsearch.index.mapper.DocumentParser.parseArray(DocumentParser.java:428)
app//org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:284)
app//org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:260)
app//org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:354)
app//org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:387)
app//org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:282)
app//org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:260)
app//org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:118)
app//org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:80)
app//org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:81)
app//org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:998)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:947)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:891)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:320)
app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:250)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base@17.0.2/java.lang.Thread.run(Thread.java:833)
100.0% [cpu=62.9%, other=37.1%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[instance-0000000001][write][T#1]'
2/10 snapshots sharing following 28 elements
app//org.apache.lucene.util.AttributeSource.clearAttributes(AttributeSource.java:274)
app//org.apache.lucene.document.Field$BinaryTokenStream.incrementToken(Field.java:515)
app//org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1142)
app//org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:733)
app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620)
app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:239)
app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1530)
app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1815)
app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1468)
app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1256)
app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1192)
app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1040)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:973)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:891)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:320)
app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:250)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base@17.0.2/java.lang.Thread.run(Thread.java:833)
3/10 snapshots sharing following 28 elements
app//org.apache.lucene.util.AttributeSource.endAttributes(AttributeSource.java:284)
app//org.apache.lucene.analysis.TokenStream.end(TokenStream.java:163)
app//org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1250)
app//org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:733)
app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620)
app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:239)
app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1530)
app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1815)
app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1468)
app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1256)
app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1192)
app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1040)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:973)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:891)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:320)
app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:250)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base@17.0.2/java.lang.Thread.run(Thread.java:833)
2/10 snapshots sharing following 25 elements
app//org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:733)
app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620)
app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:239)
app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1530)
app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1815)
app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1468)
app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1256)
app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1192)
app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1040)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:973)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:891)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:320)
app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:250)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base@17.0.2/java.lang.Thread.run(Thread.java:833)
2/10 snapshots sharing following 24 elements
app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620)
app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:239)
app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1530)
app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1815)
app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1468)
app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1256)
app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1192)
app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1040)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:973)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:891)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:320)
app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:250)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base@17.0.2/java.lang.Thread.run(Thread.java:833)
unique snapshot
java.base@17.0.2/java.util.zip.Deflater.deflateBytesBytes(Native Method)
java.base@17.0.2/java.util.zip.Deflater.deflate(Deflater.java:586)
java.base@17.0.2/java.util.zip.Deflater.deflate(Deflater.java:465)
app//org.apache.lucene.codecs.lucene90.DeflateWithPresetDictCompressionMode$DeflateWithPresetDictCompressor.doCompress(DeflateWithPresetDictCompressionMode.java:188)
app//org.apache.lucene.codecs.lucene90.DeflateWithPresetDictCompressionMode$DeflateWithPresetDictCompressor.compress(DeflateWithPresetDictCompressionMode.java:218)
app//org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.flush(Lucene90CompressingStoredFieldsWriter.java:260)
app//org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.finishDocument(Lucene90CompressingStoredFieldsWriter.java:191)
app//org.apache.lucene.index.StoredFieldsConsumer.finishDocument(StoredFieldsConsumer.java:69)
app//org.apache.lucene.index.IndexingChain.finishStoredFields(IndexingChain.java:566)
app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:632)
app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:239)
app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1530)
app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1815)
app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1468)
app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1256)
app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1192)
app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1040)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:973)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:891)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:320)
app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:250)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base@17.0.2/java.lang.Thread.run(Thread.java:833)
I had to truncate the output because it was too big to be posted all at once
100.0% [cpu=62.8%, other=37.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[instance-0000000001][write][T#2]'
3/10 snapshots sharing following 25 elements
app//org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:733)
app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620)
app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:239)
app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1530)
app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1815)
app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1468)
app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1256)
app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1192)
app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1040)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:973)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:891)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:320)
app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:250)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base@17.0.2/java.lang.Thread.run(Thread.java:833)
2/10 snapshots sharing following 25 elements
app//org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:723)
app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620)
app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:239)
app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1530)
app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1815)
app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1468)
app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1256)
app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1192)
app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1040)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:973)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:891)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:320)
app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:250)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base@17.0.2/java.lang.Thread.run(Thread.java:833)
3/10 snapshots sharing following 33 elements
java.base@17.0.2/java.util.zip.Deflater.deflateBytesBytes(Native Method)
java.base@17.0.2/java.util.zip.Deflater.deflate(Deflater.java:586)
java.base@17.0.2/java.util.zip.Deflater.deflate(Deflater.java:465)
app//org.apache.lucene.codecs.lucene90.DeflateWithPresetDictCompressionMode$DeflateWithPresetDictCompressor.doCompress(DeflateWithPresetDictCompressionMode.java:188)
app//org.apache.lucene.codecs.lucene90.DeflateWithPresetDictCompressionMode$DeflateWithPresetDictCompressor.compress(DeflateWithPresetDictCompressionMode.java:218)
app//org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.flush(Lucene90CompressingStoredFieldsWriter.java:260)
app//org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.finishDocument(Lucene90CompressingStoredFieldsWriter.java:191)
app//org.apache.lucene.index.StoredFieldsConsumer.finishDocument(StoredFieldsConsumer.java:69)
app//org.apache.lucene.index.IndexingChain.finishStoredFields(IndexingChain.java:566)
app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:632)
app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:239)
app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1530)
app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1815)
app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1468)
app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1256)
app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1192)
app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1040)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:973)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:891)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:320)
app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:250)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base@17.0.2/java.lang.Thread.run(Thread.java:833)
2/10 snapshots sharing following 25 elements
app//org.elasticsearch.index.mapper.DocumentParser.parseArray(DocumentParser.java:428)
app//org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:284)
app//org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:260)
app//org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:354)
app//org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:387)
app//org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:282)
app//org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:260)
app//org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:118)
app//org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:80)
app//org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:81)
app//org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:998)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:947)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:891)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:320)
app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:250)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base@17.0.2/java.lang.Thread.run(Thread.java:833)
It looks like a lot of time is spent writing data. What kind of storage do you have? Is it local SSDs? What does disk I/O and iowait look like on the nodes?
Hello Christian,
Thanks for helping me. Well since we're using Elastic Cloud hot nodes tier and don't have control over hardware, I guess it is fast SSDs ? How can I see Disk I/O metrics?
Philippe
Did not see you are using hot nodes in Elastic Cloud. Then disk performance might likely not be the issue.
If massive data ingestion is the issue, I would like to know how can I identify which indices are putting the cluster down to its knees? What can I modify to better balance the shards (CPU of 1 node out of 3 seems fine) or maybe reduce the load before thinking of upgrading to the next hot node tier. Cost optimisation is a one of my organization's main concern. Is there any reference out there in terms of ingestion rate versus resources?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.