# Elasticsearch Frequently Failed and Slows Down After Upgrade to 6.4 **URL:** \ **Category:** Elasticsearch\ **Created:** [September 13, 2018, 10:37am UTC](https://discuss.elastic.co/t/elasticsearch-frequently-failed-and-slows-down-after-upgrade-to-6-4/148435 "2018-09-13T10:37:07Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![fiq](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fiq](https://discuss.elastic.co/u/fiq)\ **Post date:** [September 13, 2018, 10:37am UTC](https://discuss.elastic.co/t/elasticsearch-frequently-failed-and-slows-down-after-upgrade-to-6-4/148435/1 "2018-09-13T10:37:08Z")
Hi, I have this problem where elasticsearch is frequently failed and shard allocation after restart is very slow after upgrading to 6.4 from 6.2. ``` [2018-09-13T17:26:23,546][WARN][o.e.i.c.IndicesClusterStateService] [ELK1-preprod] [[clientzone_signaturepos-2018.08.24][2]] marking and sending shard failed due to [failed recovery] org.elasticsearch.indices.recovery.RecoveryFailedException: [clientzone_signaturepos-2018.08.24][2]: Recovery failed from {ELK2-preprod}{X9uh1liyTGK2LWyMGv3BOA}{Ic15dOerSW6yGmygNTE-zw}{10.56.20.95}{10.56.20.95:9300}{ml.machine_memory=33568194560, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} into {ELK1-preprod}{ykLXYsriTPyQkkOQVKK41A}{GG-Vbx6lS1GHblvUE-QrNQ}{10.56.20.94}{10.56.20.94:9300}{ml.machine_memory=33568194560, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:282) [elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:80) [elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:623) [elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.0.jar:6.4.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] Caused by: org.elasticsearch.transport.RemoteTransportException: [ELK2-preprod][10.56.20.95:9300][internal:index/shard/recovery/start_recovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] prepare target for translog failed at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:191) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:98) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$000(PeerRecoverySourceService.java:50) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:107) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:104) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) ~[?:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) ~[?:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1605) ~[elasticsearch-6.4.0.jar:6.4.0] ... 5 more Caused by: org.elasticsearch.transport.RemoteTransportException: [ELK1-preprod][10.56.20.94:9300][internal:index/shard/recovery/prepare_translog] Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to create engine at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:199) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:160) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[elasticsearch-6.4.0.jar:6.4.0] ... 5 more [2018-09-13T17:26:34,014][WARN][o.e.g.DanglingIndicesState] [ELK1- preprod] [[clientzone_integrationconnector-2018.07.16/ubGfO0sGSCqV8LZxmkxMVQ]] can not be imported as a dangling index, as index with same name already exists in cluster metadata [2018-09-13T17:26:34,014][WARN][o.e.g.DanglingIndicesState] [ELK1-preprod] [[error-partnerzone_partner-2018.07.16/fzlLkW57T6iZfTUzAhDgtw]] can not be imported as a dangling index, as index with same name already exists in cluster metadata ``` Its been 7 hour and active\_shards\_percent\_as\_number is only 59% and cluster health is red. Can someone give me advice on how to resolve this problem? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 13, 2018, 10:54am UTC](https://discuss.elastic.co/t/elasticsearch-frequently-failed-and-slows-down-after-upgrade-to-6-4/148435/2 "2018-09-13T10:54:02Z")
What is the output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-stats.html)? How many indices/shards do you have in the cluster? ---
**Author:** ![fiq](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@fiq](https://discuss.elastic.co/u/fiq)\ **Post date:** [September 13, 2018, 11:20am UTC](https://discuss.elastic.co/t/elasticsearch-frequently-failed-and-slows-down-after-upgrade-to-6-4/148435/3 "2018-09-13T11:20:09Z")
Hi, One of my none just failed with this error : ``` [2018-09-13T17:40:05,492][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ELK2-preprod] fatal error in thread [elasticsearch[ELK2-preprod][generic][T#3]], exitingjava.lang.StackOverflowError: null at org.elasticsearch.common.io.stream.BytesStreamOutput.writeBytes(BytesStreamOutput.java:89) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.common.io.stream.StreamOutput.writeBytes(StreamOutput.java:172) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.common.io.stream.StreamOutput.writeString(StreamOutput.java:399) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.common.io.stream.StreamOutput.writeOptionalString(StreamOutput.java:320) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:937) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.ElasticsearchException.writeStackTraces(ElasticsearchException.java:743) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:942) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:940) ~[elasticsearch-6.4.0.jar:6.4.0] at org.elasticsearch.ElasticsearchException.writeStackTraces(ElasticsearchException.java:743) ~[elasticsearch-6.4.0.jar:6.4.0] ``` As for cluster stat, here it is : ``` { "_nodes" : { "total" : 2, "successful" : 2, "failed" : 0 }, "cluster_name" : "elastic-preprod", "timestamp" : 1536837474401, "status" : "red", "indices" : { "count" : 441, "shards" : { "total" : 2134, "primaries" : 2123, "replication" : 0.0051813471502590676, "index" : { "shards" : { "min" : 1, "max" : 9, "avg" : 4.839002267573696 }, "primaries" : { "min" : 1, "max" : 5, "avg" : 4.814058956916099 }, "replication" : { "min" : 0.0, "max" : 1.0, "avg" : 0.007709750566893424 } } }, "docs" : { "count" : 165795439, "deleted" : 392243 }, "store" : { "size" : "21.3gb", "size_in_bytes" : 22957601319 }, "fielddata" : { "memory_size" : "0b", "memory_size_in_bytes" : 0, "evictions" : 0 }, "query_cache" : { "memory_size" : "0b", "memory_size_in_bytes" : 0, "total_count" : 0, "hit_count" : 0, "miss_count" : 0, "cache_size" : 0, "cache_count" : 0, "evictions" : 0 }, "completion" : { "size" : "0b", "size_in_bytes" : 0 }, "segments" : { "count" : 9162, "memory" : "178.8mb", "memory_in_bytes" : 187552499, "terms_memory" : "144.8mb", "terms_memory_in_bytes" : 151834956, "stored_fields_memory" : "9.7mb", "stored_fields_memory_in_bytes" : 10178872, "term_vectors_memory" : "0b", "term_vectors_memory_in_bytes" : 0, "norms_memory" : "11.2mb", "norms_memory_in_bytes" : 11792512, "points_memory" : "7.5mb", "points_memory_in_bytes" : 7967983, "doc_values_memory" : "5.5mb", "doc_values_memory_in_bytes" : 5778176, "index_writer_memory" : "0b", "index_writer_memory_in_bytes" : 0, "version_map_memory" : "0b", "version_map_memory_in_bytes" : 0, "fixed_bit_set" : "332kb", "fixed_bit_set_memory_in_bytes" : 340032, "max_unsafe_auto_id_timestamp" : 1536834850703, "file_sizes" : { } } }, "nodes" : { "count" : { "total" : 2, "data" : 2, "coordinating_only" : 0, "master" : 2, "ingest" : 2 }, "versions" : [ "6.4.0" ], "os" : { "available_processors" : 16, "allocated_processors" : 16, "names" : [ { "name" : "Linux", "count" : 2 } ], "mem" : { "total" : "62.5gb", "total_in_bytes" : 67136389120, "free" : "1.6gb", "free_in_bytes" : 1794719744, "used" : "60.8gb", "used_in_bytes" : 65341669376, "free_percent" : 3, "used_percent" : 97 } }, "process" : { "cpu" : { "percent" : 17 }, "open_file_descriptors" : { "min" : 9053, "max" : 9783, "avg" : 9418 } }, "jvm" : { "max_uptime" : "9.9m", "max_uptime_in_millis" : 599770, "versions" : [ { "version" : "1.8.0_161", "vm_name" : "Java HotSpot(TM) 64-Bit Server VM", "vm_version" : "25.161-b12", "vm_vendor" : "Oracle Corporation", "count" : 2 } ], "mem" : { "heap_used" : "2.6gb", "heap_used_in_bytes" : 2807195992, "heap_max" : "27.8gb", "heap_max_in_bytes" : 29925310464 }, "threads" : 182 }, "fs" : { "total" : "393.4gb", "total_in_bytes" : 422487998464, "free" : "224.2gb", "free_in_bytes" : 240744771584, "available" : "208.1gb", "available_in_bytes" : 223445872640 }, "plugins" : [], "network_types" : { "transport_types" : { "security4" : 2 }, "http_types" : { "security4" : 2 } } } } ``` Thanks in advance! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 13, 2018, 11:27am UTC](https://discuss.elastic.co/t/elasticsearch-frequently-failed-and-slows-down-after-upgrade-to-6-4/148435/4 "2018-09-13T11:27:57Z")
That is a very large number of indices and shards given the data volume you have, which as you see will cause problems. Please read [this blog post for some guidance on shards and sharding](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster), and then try to reduce the number of shards you use dramatically, e.g. by reindexing your data. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 11, 2018, 11:31am UTC](https://discuss.elastic.co/t/elasticsearch-frequently-failed-and-slows-down-after-upgrade-to-6-4/148435/6 "2018-10-11T11:31:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.