Index document: very bad performance in Elasticsearch 8.2.0

Alexander_Ott · May 23, 2022, 11:34am

We are trying to migrate from Elasticsearch 6.7.1 to Elasticsearch 8.2.0.

With the same amount of documents per bulk request, we always get a socket timeout in Elasticsearch 8.2.0. Everything works fine in Elasticsearch 6.7.1.

I noticed that indexing the same document in Elasticsearch 8.2.0 takes 44152ms. In Elasticsearch 6.7.1, indexing takes 50ms. Are there any known issues for Elasticsearch 8.2.0?

I created an issue for this problem --> bad indexing performance in elasticsearch 8.2.0 · Issue #87036 · elastic/elasticsearch · GitHub

system · May 23, 2022, 11:34am

Elasticsearch 6.7.1 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

Christian_Dahlqvist · May 23, 2022, 4:23pm

Are the two clusters using exactly the same hardware? What is the specification of the clusters? What does the hot threads API show while the longer query executes?

Alexander_Ott · May 24, 2022, 5:07am

It's just a single node for dev setup.

You can reproduce it with the requests which i added to bad indexing performance in elasticsearch 8.2.0 · Issue #87036 · elastic/elasticsearch · GitHub

Hot Threads:

::: {es01}{OxiiGQ59StqP874DgpJLyQ}{pI5Cc7EETOSYDOPKiQTZvw}{172.31.0.2}{172.31.0.2:9300}{dm}{xpack.installed=true}
   Hot threads at 2022-05-23T14:52:47.572Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   101.0% [cpu=101.0%, other=0.0%] (505.1ms out of 500ms) cpu usage by thread 'elasticsearch[es01][write][T#3]'
     3/10 snapshots sharing following 28 elements
       app//org.apache.lucene.util.UnicodeUtil.UTF16toUTF8(UnicodeUtil.java:174)
       app//org.apache.lucene.util.BytesRefBuilder.copyChars(BytesRefBuilder.java:141)
       app//org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl.getBytesRef(CharTermAttributeImpl.java:93)
       app//org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1224)
       app//org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:733)
       app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620)
       app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:241)
       app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
       app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1531)
       app//org.apache.lucene.index.IndexWriter.softUpdateDocuments(IndexWriter.java:1576)
       app//org.elasticsearch.index.engine.InternalEngine.updateDocs(InternalEngine.java:1375)
       app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1188)
       app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
       app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1039)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:972)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:890)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:319)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:251)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
       app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@18/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
       java.base@18/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
       java.base@18/java.lang.Thread.run(Thread.java:833)
     2/10 snapshots sharing following 25 elements
       app//org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1138)
       app//org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:733)
       app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620)
       app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:241)
       app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
       app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1531)
       app//org.apache.lucene.index.IndexWriter.softUpdateDocuments(IndexWriter.java:1576)
       app//org.elasticsearch.index.engine.InternalEngine.updateDocs(InternalEngine.java:1375)
       app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1188)
       app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
       app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1039)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:972)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:890)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:319)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:251)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
       app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@18/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
       java.base@18/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
       java.base@18/java.lang.Thread.run(Thread.java:833)
     5/10 snapshots sharing following 23 elements
       app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620)
       app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:241)
       app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
       app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1531)
       app//org.apache.lucene.index.IndexWriter.softUpdateDocuments(IndexWriter.java:1576)
       app//org.elasticsearch.index.engine.InternalEngine.updateDocs(InternalEngine.java:1375)
       app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1188)
       app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:995)
       app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1039)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:972)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:890)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:319)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:185)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:251)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:131)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:70)
       app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:210)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@18/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
       java.base@18/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
       java.base@18/java.lang.Thread.run(Thread.java:833)

system · June 21, 2022, 5:07am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Extremely slow indexing -- java throwing http excetion errors Elasticsearch	2	686	July 6, 2017
Elasticsearch poor indexing performance Elasticsearch	6	844	December 1, 2017
Performance reduction in elasticsearch 8.0.0 Elasticsearch	6	2033	March 13, 2022
Upgrade to elasticsearch 2.3.3 from 2.3.1 - Slow Indexing Elasticsearch	6	1565	July 4, 2016
Slow performance when switching from elasticsearch c 1.4 -> 2.0 Elasticsearch	6	427	July 5, 2017

Index document: very bad performance in Elasticsearch 8.2.0

Related Topics