Index document: very bad performance in Elasticsearch 8.2.0

Alexander_Ott · May 23, 2022, 11:34am

We are trying to migrate from Elasticsearch 6.7.1 to Elasticsearch 8.2.0.

With the same amount of documents per bulk request, we always get a socket timeout in Elasticsearch 8.2.0. Everything works fine in Elasticsearch 6.7.1.

I noticed that indexing the same document in Elasticsearch 8.2.0 takes 44152ms. In Elasticsearch 6.7.1, indexing takes 50ms. Are there any known issues for Elasticsearch 8.2.0?

I created an issue for this problem --> bad indexing performance in elasticsearch 8.2.0 · Issue #87036 · elastic/elasticsearch · GitHub

system · May 23, 2022, 11:34am

Elasticsearch 6.7.1 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

Christian_Dahlqvist · May 23, 2022, 4:23pm

Are the two clusters using exactly the same hardware? What is the specification of the clusters? What does the hot threads API show while the longer query executes?

Alexander_Ott · May 24, 2022, 5:07am

It's just a single node for dev setup.

You can reproduce it with the requests which i added to bad indexing performance in elasticsearch 8.2.0 · Issue #87036 · elastic/elasticsearch · GitHub

Hot Threads:

::: {es01}{OxiiGQ59StqP874DgpJLyQ}{pI5Cc7EETOSYDOPKiQTZvw}{172.31.0.2}{172.31.0.2:9300}{dm}{xpack.installed=true}
   Hot threads at 2022-05-23T14:52:47.572Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   101.0% [cpu=101.0%, other=0.0%] (505.1ms out of 500ms) cpu usage by thread 'elasticsearch[es01][write][T#3]'
     3/10 snapshots sharing following 28 elements
       app//org.apache.lucene.util.UnicodeUtil.UTF16toUTF8(UnicodeUtil.java:174)
       app//org.apache.lucene.util.BytesRefBuilder.copyChars(BytesRefBuilder.java:141)
       app//org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl.getBytesRef(CharTermAttributeImpl.java:93)
       app//org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1224)
       app//org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:733)
       app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620)
       app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:241)
       app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
       app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1531)
       app//org.apache.lucene.index.IndexWriter.softUpdateDocuments(IndexWriter.java:1576)
       app//org.elasticsearch.index.engine.InternalEngine.updateDocs(InternalEngine.java:1375)
       app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1188)
       app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
       app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1039)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:972)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:890)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:319)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:251)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
       app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@18/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
       java.base@18/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
       java.base@18/java.lang.Thread.run(Thread.java:833)
     2/10 snapshots sharing following 25 elements
       app//org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1138)
       app//org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:733)
       app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620)
       app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:241)
       app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
       app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1531)
       app//org.apache.lucene.index.IndexWriter.softUpdateDocuments(IndexWriter.java:1576)
       app//org.elasticsearch.index.engine.InternalEngine.updateDocs(InternalEngine.java:1375)
       app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1188)
       app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
       app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1039)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:972)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:890)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:319)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:251)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
       app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@18/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
       java.base@18/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
       java.base@18/java.lang.Thread.run(Thread.java:833)
     5/10 snapshots sharing following 23 elements
       app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620)
       app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:241)
       app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
       app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1531)
       app//org.apache.lucene.index.IndexWriter.softUpdateDocuments(IndexWriter.java:1576)
       app//org.elasticsearch.index.engine.InternalEngine.updateDocs(InternalEngine.java:1375)
       app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1188)
       app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
       app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1039)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:972)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:890)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:319)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:251)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
       app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@18/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
       java.base@18/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
       java.base@18/java.lang.Thread.run(Thread.java:833)

system · June 21, 2022, 5:07am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.