# Elasticsearch 1.4.4 sporatically crashes **URL:** \ **Category:** Elasticsearch\ **Created:** [April 5, 2017, 5:37pm UTC](https://discuss.elastic.co/t/elasticsearch-1-4-4-sporatically-crashes/81351 "2017-04-05T17:37:32Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![dmcpherson](https://avatars.discourse-cdn.com/v4/letter/d/6bbea6/32.png) [@dmcpherson](https://discuss.elastic.co/u/dmcpherson)\ **Post date:** [April 5, 2017, 5:37pm UTC](https://discuss.elastic.co/t/elasticsearch-1-4-4-sporatically-crashes/81351/1 "2017-04-05T17:37:32Z")
I have a single node instance of Elastic search 1.4.4 running and it has been sporadically crashing and I have not been able to identify the root cause. Here are the stats for my server and for my elasticsearch instance: java - jdk1.8.0\_25 ElasticSearch - 1.4.4 ES\_HEAP\_SIZE=16g Marvel - 1.1.0 It had been running fine up until last weekend (4/2/2017) when all of the sudden it started crashing after 1-3 hours of up time. I have a couple suspects but I can not pin point what the root cause is. 1. The server is currently under a heavy load. Could be that our hardware can't handle it. 2. We have one extremely large index, 257 million documents in total I am not sure if this could cause problems. 3. Something has caused our indexes to become corrupted, judging from the logs which I have attached. Marvel particular appears to be having trouble creating any indexes, getting repeated UnavailableShardsException 4. We have over 10,000 shards, with 5436 active and 5446 replicas for just of 1000 indexes. I will post an abbreviated log from the last crash below this. Any info that could point me in the right direction would be greatly appreciated. Can provide more info if needed. ---
**Author:** ![dmcpherson](https://avatars.discourse-cdn.com/v4/letter/d/6bbea6/32.png) [@dmcpherson](https://discuss.elastic.co/u/dmcpherson)\ **Post date:** [April 5, 2017, 5:47pm UTC](https://discuss.elastic.co/t/elasticsearch-1-4-4-sporatically-crashes/81351/2 "2017-04-05T17:47:42Z")
[2017-04-05 12:47:18,893][INFO][node] [ki003] version[1.4.4], pid[28350], build[c88f77f/2015-02-19T13:05:36Z] [2017-04-05 12:47:18,893][INFO][node] [ki003] initializing ... [2017-04-05 12:47:18,914][INFO][plugins] [ki003] loaded [marvel], sites [marvel, head, migration] [2017-04-05 12:47:20,403][WARN][monitor.jvm] [ki003] ignoring gc\_threshold for [young], missing warn/info/debug values [2017-04-05 12:47:20,404][WARN][monitor.jvm] [ki003] ignoring gc\_threshold for [old], missing warn/info/debug values [2017-04-05 12:48:00,419][INFO][node] [ki003] initialized [2017-04-05 12:48:00,419][INFO][node] [ki003] starting ... [2017-04-05 12:48:00,568][INFO][transport] [ki003] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/172.18.146.131:9300]} [2017-04-05 12:48:00,586][INFO][discovery] [ki003] elasticsearch\_intellinx\_prod/Z48wavFPSkONsN3T5mPacA [2017-04-05 12:48:04,363][INFO][cluster.service] [ki003] new\_master [ki003][Z48wavFPSkONsN3T5mPacA][intxpprki003][inet[/172.18.146.131:9300]], reason: zen-disco-join (elected\_as\_master) [2017-04-05 12:48:04,658][INFO][http] [ki003] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/172.18.146.131:9200]} [2017-04-05 12:48:04,658][INFO][node] [ki003] started [2017-04-05 12:48:06,361][DEBUG][action.search.type] [ki003] All shards failed for phase: [query] [2017-04-05 12:48:06,442][DEBUG][action.search.type] [ki003] All shards failed for phase: [query] [2017-04-05 12:48:07,214][DEBUG][action.search.type] [ki003] All shards failed for phase: [query] [2017-04-05 12:48:07,267][DEBUG][action.search.type] [ki003] All shards failed for phase: [query] [2017-04-05 12:48:07,685][DEBUG][action.search.type] [ki003] All shards failed for phase: [query] [2017-04-05 12:48:07,738][DEBUG][action.search.type] [ki003] All shards failed for phase: [query] [2017-04-05 12:48:07,799][DEBUG][action.search.type] [ki003] All shards failed for phase: [query] [2017-04-05 12:48:07,925][DEBUG][action.search.type] [ki003] All shards failed for phase: [query] [2017-04-05 12:48:07,969][INFO][gateway] [ki003] recovered [1089] indices into cluster\_state [2017-04-05 12:48:07,970][DEBUG][action.admin.indices.create] [ki003] [efax\_consumers] failed to create org.elasticsearch.indices.InvalidIndexNameException: [efax\_consumers] Invalid index name [efax\_consumers], already exists as alias at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validateIndexName(MetaDataCreateIndexService.java:197) at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validate(MetaDataCreateIndexService.java:559) at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.access$200(MetaDataCreateIndexService.java:87) at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$2.execute(MetaDataCreateIndexService.java:243) at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:352) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:184) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:154) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [2017-04-05 12:48:07,976][DEBUG][action.search.type] [ki003] All shards failed for phase: [query] [2017-04-05 12:48:07,979][DEBUG][action.search.type] [ki003] All shards failed for phase: [query] .... [2017-04-05 12:50:11,761][ERROR][marvel.agent.exporter] [ki003] create failure (index:[.marvel-2017.04.05] type: [node\_event]): UnavailableShardsException[[.marvel-2017.04.05][0] Primary shard is not active or isn't assigned is a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@376fe9b7] [2017-04-05 12:50:11,761][ERROR][marvel.agent.exporter] [ki003] create failure (index:[.marvel-2017.04.05] type: [cluster\_event]): UnavailableShardsException[[.marvel-2017.04.05][0] Primary shard is not active or isn't assigned is a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@376fe9b7] [2017-04-05 12:50:11,761][ERROR][marvel.agent.exporter] [ki003] create failure (index:[.marvel-2017.04.05] type: [shard\_event]): UnavailableShardsException[[.marvel-2017.04.05][0] Primary shard is not active or isn't assigned is a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@376fe9b7] [2017-04-05 12:50:11,761][ERROR][marvel.agent.exporter] [ki003] create failure (index:[.marvel-2017.04.05] type: [shard\_event]): UnavailableShardsException[[.marvel-2017.04.05][0] Primary shard is not active or isn't assigned is a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@376fe9b7] .... [2017-04-05 12:56:56,982][DEBUG][action.bulk] [ki003] observer timed out. notifying listener. timeout setting [1m], time since start [1m] [2017-04-05 12:59:10,804][DEBUG][action.bulk] [ki003] observer timed out. notifying listener. timeout setting [1m], time since start [1m] [2017-04-05 12:59:10,804][DEBUG][action.bulk] [ki003] observer timed out. notifying listener. timeout setting [1m], time since start [1m] [2017-04-05 12:59:10,804][DEBUG][action.bulk] [ki003] observer timed out. notifying listener. timeout setting [1m], time since start [1m] [2017-04-05 12:59:10,805][DEBUG][action.bulk] [ki003] observer timed out. notifying listener. timeout setting [1m], time since start [1m] [2017-04-05 12:59:10,805][DEBUG][action.bulk] [ki003] observer timed out. notifying listener. timeout setting [1m], time since start [1m] [2017-04-05 13:00:12,227][DEBUG][action.bulk] [ki003] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2017-04-05 13:00:12,227][DEBUG][action.bulk] [ki003] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2017-04-05 13:00:12,264][DEBUG][action.bulk] [ki003] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2017-04-05 13:00:12,264][DEBUG][action.bulk] [ki003] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [April 6, 2017, 7:41am UTC](https://discuss.elastic.co/t/elasticsearch-1-4-4-sporatically-crashes/81351/3 "2017-04-06T07:41:12Z")
Hey, this is very hard to tell with this kind of information. Also I am not too sure how much sense it makes to debug further issues, given that you have 10k shards (ok, 'only' 5k shards) on a single node, which you shouldnt do at all. Plus you are using a really old version of Elasticsearch, released more than two years ago. You should check stuff like node stats, node infos, garbage collection, hot threads and pinpoint down the issue further, but even if you know it, you need to invest some time to get a stable system. --Alex ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 4, 2017, 7:41am UTC](https://discuss.elastic.co/t/elasticsearch-1-4-4-sporatically-crashes/81351/4 "2017-05-04T07:41:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.