# Elasticsearch upgrade from elasticsearch 5.1.2 to 5.6.14, index operation tps rise cause bulk rejected **URL:** \ **Category:** Elasticsearch\ **Created:** [May 24, 2019, 9:00am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-elasticsearch-5-1-2-to-5-6-14-index-operation-tps-rise-cause-bulk-rejected/182594 "2019-05-24T09:00:34Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![songfj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/songfj/32/22256_2.png) [@songfj](https://discuss.elastic.co/u/songfj)\ **Post date:** [May 24, 2019, 9:00am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-elasticsearch-5-1-2-to-5-6-14-index-operation-tps-rise-cause-bulk-rejected/182594/1 "2019-05-24T09:00:34Z")
my bulk queue size = 200 I find the bulk queue always reject like follow, I find the rejected size can't reduce hot thread like fllow: $ curl localhost:8200/\_cat/thread\_pool/bulk?h=node\_id,name,queue,queue\_size,rejected ` eYdTntK_xxxxxxdw-nmkw bulk 0 200 193083 31VHf-MxxxxxxtIagxUNCQ bulk 0 200 0 FH7Ly3Svxxxxxx4UFR-ag bulk 0 200 52 0HtssJxxxxxx4cCQBTHQ bulk 0 200 0 Tiq_ELaUxxxxxxm7IhYD_Q bulk 0 200 145 Bg4oFea0xxxxxx3cGY4sboQ bulk 0 200 2708186 6nNHxyh2xxxxxxb0zlMpZQ bulk 0 200 32139 B3AM2fExxxxxxvhuqlvzg bulk 0 200 0 y7OTpxxxxxx9QdqpMDew bulk 0 200 24 DoXAsxxxxxxQVb3IgOu6RA bulk 0 200 0 has6UOxxxxxxxHMzE7DuAJg bulk 0 200 85 u5SfGUzxxxxxxoqxKn1Q bulk 0 200 0 Ru4BcxxxxxxK6vHhPGg bulk 0 200 0 ` ``` $ curl localhost:9200/_nodes/Bg4oFea0xxxxxxxGY4sboQ/hot_threads ``` ::: {127.0.0.1}{Bg4oFeaxxxxxxV3cGY4sboQ}{ceDy3FuxRxxxxxUWNhA}{127.0.0.x}{127.0.0.x:9300}{storetype=ssd} Hot threads at 2019-05-24T07:17:19.700Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: 101.1% (505.4ms out of 500ms) cpu usage by thread 'elasticsearch[xxxxxxxxx][bulk][T#27]' 10/10 snapshots sharing following 35 elements java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry(ThreadLocal.java:617) java.lang.ThreadLocal$ThreadLocalMap.replaceStaleEntry(ThreadLocal.java:575) java.lang.ThreadLocal$ThreadLocalMap.set(ThreadLocal.java:476) java.lang.ThreadLocal$ThreadLocalMap.access$100(ThreadLocal.java:298) java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:184) java.lang.ThreadLocal.get(ThreadLocal.java:170) org.apache.lucene.util.CloseableThreadLocal.get(CloseableThreadLocal.java:78) org.elasticsearch.common.lucene.uid.VersionsResolver.getLookupState(VersionsResolver.java:72) org.elasticsearch.common.lucene.uid.VersionsResolver.loadDocIdAndVersion(VersionsResolver.java:120) org.elasticsearch.common.lucene.uid.VersionsResolver.loadVersion(VersionsResolver.java:137) org.elasticsearch.index.engine.InternalEngine.loadCurrentVersionFromIndex(InternalEngine.java:1377) org.elasticsearch.index.engine.InternalEngine.resolveDocVersion(InternalEngine.java:393) org.elasticsearch.index.engine.InternalEngine.compareOpToLuceneDocBasedOnVersions(InternalEngine.java:408) org.elasticsearch.index.engine.InternalEngine.planIndexingAsNonPrimary(InternalEngine.java:545) org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:496) org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:557) org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546) org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnReplica(TransportShardBulkAction.java:449) org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:383) org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:69) org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:522) org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:491) org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:151) org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationLock(IndexShard.java:1675) org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:594) org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:475) org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:464) org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1556) org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) 101.1% (505.4ms out of 500ms) cpu usage by thread 'elasticsearch[xxxxx.xxxx.xxxx][bulk][T#29]' 10/10 snapshots sharing following 41 elements java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry(ThreadLocal.java:617) java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:440) java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419) java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298) java.lang.ThreadLocal.get(ThreadLocal.java:163) org.apache.lucene.util.CloseableThreadLocal.get(CloseableThreadLocal.java:78) org.elasticsearch.common.lucene.uid.VersionsResolver.getLookupState(VersionsResolver.java:72) org.elasticsearch.common.lucene.uid.VersionsResolver.loadDocIdAndVersion(VersionsResolver.java:120) org.elasticsearch.common.lucene.uid.VersionsResolver.loadVersion(VersionsResolver.java:137) org.elasticsearch.index.engine.InternalEngine.loadCurrentVersionFromIndex(InternalEngine.java:1377) org.elasticsearch.index.engine.InternalEngine.resolveDocVersion(InternalEngine.java:393) org.elasticsearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:570) org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:493) org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:557) org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546) ++++++++++++++++++++++++++++++++++++++= I find one node rejected always great than 2706146 Thanks ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 24, 2019, 10:12am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-elasticsearch-5-1-2-to-5-6-14-index-operation-tps-rise-cause-bulk-rejected/182594/2 "2019-05-24T10:12:40Z")
Please don't post images of text as they are hardly readable and not searchable. Instead paste the text and format it with `` icon. Check the preview window. ---
**Author:** ![songfj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/songfj/32/22256_2.png) [@songfj](https://discuss.elastic.co/u/songfj)\ **Post date:** [May 24, 2019, 11:00am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-elasticsearch-5-1-2-to-5-6-14-index-operation-tps-rise-cause-bulk-rejected/182594/3 "2019-05-24T11:00:59Z")
org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:493) org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:145) org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:114) org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:69) org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:975) org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:944) org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:345) org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:270) org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:924) org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:921) org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:151) org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1659) org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:933) org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:92) org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:291) org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:266) org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:248) org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:662) org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) 101.1% (505.3ms out of 500ms) cpu usage by thread 'elasticsearch[xxxx.xxxxxx][bulk][T#12]' 10/10 snapshots sharing following 45 elements java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry(ThreadLocal.java:617) java.lang.ThreadLocal$ThreadLocalMap.replaceStaleEntry(ThreadLocal.java:575) java.lang.ThreadLocal$ThreadLocalMap.set(ThreadLocal.java:476) java.lang.ThreadLocal$ThreadLocalMap.access$100(ThreadLocal.java:298) java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:184) java.lang.ThreadLocal.get(ThreadLocal.java:170) org.apache.lucene.util.CloseableThreadLocal.get(CloseableThreadLocal.java:78) org.apache.lucene.index.CodecReader.getNumericDocValues(CodecReader.java:150) org.apache.lucene.index.FilterLeafReader.getNumericDocValues(FilterLeafReader.java:436) org.elasticsearch.common.lucene.uid.PerThreadIDAndVersionLookup.(PerThreadIDAndVersionLookup.java:73) org.elasticsearch.common.lucene.uid.VersionsResolver.getLookupState(VersionsResolver.java:74) org.elasticsearch.common.lucene.uid.VersionsResolver.loadDocIdAndVersion(VersionsResolver.java:120) org.elasticsearch.common.lucene.uid.VersionsResolver.loadVersion(VersionsResolver.java:137) org.elasticsearch.index.engine.InternalEngine.loadCurrentVersionFromIndex(InternalEngine.java:1377) org.elasticsearch.index.engine.InternalEngine.resolveDocVersion(InternalEngine.java:393) org.elasticsearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:570) org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:493) org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:557) org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546) ---
**Author:** ![songfj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/songfj/32/22256_2.png) [@songfj](https://discuss.elastic.co/u/songfj)\ **Post date:** [May 24, 2019, 11:01am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-elasticsearch-5-1-2-to-5-6-14-index-operation-tps-rise-cause-bulk-rejected/182594/4 "2019-05-24T11:01:10Z")
org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:493) org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:145) org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:114) org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:69) org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:975) org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:944) org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:345) org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:270) org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:924) org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:921) org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:151) org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1659) org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:933) org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:92) org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:291) org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:266) org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:248) org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:662) org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) ---
**Author:** ![songfj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/songfj/32/22256_2.png) [@songfj](https://discuss.elastic.co/u/songfj)\ **Post date:** [May 24, 2019, 11:02am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-elasticsearch-5-1-2-to-5-6-14-index-operation-tps-rise-cause-bulk-rejected/182594/5 "2019-05-24T11:02:37Z")
Sorry, But the stacktrack is too large than the limit ---
**Author:** ![songfj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/songfj/32/22256_2.png) [@songfj](https://discuss.elastic.co/u/songfj)\ **Post date:** [May 24, 2019, 11:04am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-elasticsearch-5-1-2-to-5-6-14-index-operation-tps-rise-cause-bulk-rejected/182594/6 "2019-05-24T11:04:06Z")
there are another hot\_thread $ curl localhost:9200/\_nodes/Bg4oFea0QuxxxxxxboQ/hot\_threads ::: {xxxxxx-es1}{Bg4oFea0xxxxxxoQ}{ceDy3xxxxxxtVsoTKUWNhA}{xxxxxx}{xxxxxxx:9300}{storetype=ssd} Hot threads at 2019-05-24T08:30:01.431Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: 101.0% (505.2ms out of 500ms) cpu usage by thread 'elasticsearch[xxxxxxxx-es1][management][T#4]' 4/10 snapshots sharing following 32 elements java.security.AccessController.doPrivileged(Native Method) java.io.FilePermission.init(FilePermission.java:203) java.io.FilePermission.(FilePermission.java:277) java.lang.SecurityManager.checkRead(SecurityManager.java:888) sun.nio.fs.UnixPath.checkRead(UnixPath.java:795) sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49) sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144) sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) java.nio.file.Files.readAttributes(Files.java:1737) java.nio.file.Files.size(Files.java:2332) org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243) org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67) org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67) org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1402) org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1391) org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1378) org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54) org.elasticsearch.index.store.Store.stats(Store.java:332) org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:703) org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:177) org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:104) org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:53) org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1556) org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) 6/10 snapshots sharing following 18 elements org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1391) org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1378) org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54) org.elasticsearch.index.store.Store.stats(Store.java:332) org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:703) org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:177) org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:104) org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:53) org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262) org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258) org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1556) org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) ``` 8.5% (42.6ms out of 500ms) cpu usage by thread 'elasticsearch[xxxxxx-xxxx-es1][bulk][T#13]' unique snapshot java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444) java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419) java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298) java.lang.ThreadLocal.get(ThreadLocal.java:163) org.apache.lucene.util.CloseableThreadLocal.get(CloseableThreadLocal.java:78) org.elasticsearch.common.lucene.uid.VersionsResolver.getLookupState(VersionsResolver.java:72) org.elasticsearch.common.lucene.uid.VersionsResolver.loadDocIdAndVersion(VersionsResolver.java:120) org.elasticsearch.common.lucene.uid.VersionsResolver.loadVersion(VersionsResolver.java:137) org.elasticsearch.index.engine.InternalEngine.loadCurrentVersionFromIndex(InternalEngine.java:1377) org.elasticsearch.index.engine.InternalEngine.resolveDocVersion(InternalEngine.java:393) org.elasticsearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:570) org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:493) org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:557) org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546) org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:493) org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:145) org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:114) org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:69) ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 24, 2019, 11:11am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-elasticsearch-5-1-2-to-5-6-14-index-operation-tps-rise-cause-bulk-rejected/182594/7 "2019-05-24T11:11:06Z")
It looks like the nodes are very busy processing bulk requests. How many indices/shards do you have in the cluster? How many of these are you actively indexing into? Are these evenly distributed across the nodes in the cluster? What is your use-case? I would also recommend you look at the following blog posts: > **[Why am I seeing bulk rejections in my Elasticsearch cluster?](https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster)** > > Experiencing bulk rejections leading to poor indexing performance? This blog post explains why Elasticsearch rejects requests and how to get around it. > **[How many shards should I have in my Elasticsearch cluster?](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)** > > If you are looking for practical guidelines around how many indices and shards to have in your cluster, this blog post will help you avoid common pitfalls. ---
**Author:** ![songfj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/songfj/32/22256_2.png) [@songfj](https://discuss.elastic.co/u/songfj)\ **Post date:** [May 24, 2019, 11:19am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-elasticsearch-5-1-2-to-5-6-14-index-operation-tps-rise-cause-bulk-rejected/182594/8 "2019-05-24T11:19:34Z")
> [@Christian\_Dahlqvist](#): > > you look at the following blog posts The cluster detail: 8 data node machine,2 es instances in it mem: 32G/(every data node instance) 3 master node 2 client node 248 indices 3208 shards 8,195,271,243 docs ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/6/168587fb07a14a16b414c249ba6611867ef38f9f.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/9/d9c567ec54a53f1a45aee26fa6ef4ae598ee3fca.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/7/57e41c28c30d1c0818bd306ab0ce1c055b7ff026.png) ---
**Author:** ![songfj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/songfj/32/22256_2.png) [@songfj](https://discuss.elastic.co/u/songfj)\ **Post date:** [May 24, 2019, 11:27am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-elasticsearch-5-1-2-to-5-6-14-index-operation-tps-rise-cause-bulk-rejected/182594/9 "2019-05-24T11:27:34Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/6/e/6ed594630bc09dc2e71da4a77489812b60ac3b2a.png) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 24, 2019, 11:28am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-elasticsearch-5-1-2-to-5-6-14-index-operation-tps-rise-cause-bulk-rejected/182594/10 "2019-05-24T11:28:22Z")
How many of these are you actively indexing into? Are bulk requests potentially indexing into a large number of shards? ---
**Author:** ![songfj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/songfj/32/22256_2.png) [@songfj](https://discuss.elastic.co/u/songfj)\ **Post date:** [May 24, 2019, 11:40am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-elasticsearch-5-1-2-to-5-6-14-index-operation-tps-rise-cause-bulk-rejected/182594/11 "2019-05-24T11:40:13Z")
Yes I have 100 big index ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/2/f232dd84ed96e6d86553928a0bbf1fbc2840bbfe.png) A few days ago， I do a reindex for the cluster, from one to another with \_reindex api, But it reject, I have the same another cluster, that cluster es version is es 5.1.2, but that cluster doesn't \* frequently cause the rejected My all 248 indices have indexing, but the 100indices larger ---
**Author:** ![songfj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/songfj/32/22256_2.png) [@songfj](https://discuss.elastic.co/u/songfj)\ **Post date:** [May 24, 2019, 11:47am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-elasticsearch-5-1-2-to-5-6-14-index-operation-tps-rise-cause-bulk-rejected/182594/12 "2019-05-24T11:47:34Z")
We need to update our elasticsearch cluster from 5.1.2 ---\> 5.6 ----\> 6.x So we use the same config for 5.1 and 5.6 the upgrade 5.6 cluster only have index operation doesn't have search operation ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 21, 2019, 11:47am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-elasticsearch-5-1-2-to-5-6-14-index-operation-tps-rise-cause-bulk-rejected/182594/13 "2019-06-21T11:47:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.