# Unable to start Elasticsearch and Cluster is down **URL:** \ **Category:** Elasticsearch\ **Created:** [April 28, 2016, 5:34pm UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-and-cluster-is-down/48716 "2016-04-28T17:34:00Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\ **Post date:** [April 28, 2016, 5:34pm UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-and-cluster-is-down/48716/1 "2016-04-28T17:34:00Z")
I have a three node cluster and one of the nodes in the cluster is failing to start the ES services. Due to this one node not starting the whole cluster is no longer running. When attempting to start this service I am seeing the following errors in /var/log/messages: ``` Apr 28 10:27:29 p-es-1 systemd: Starting Elasticsearch... Apr 28 10:27:29 p-es-1 systemd: Started Elasticsearch. Apr 28 10:27:29 p-es-1 elasticsearch: [2016-04-28 10:27:29,980][WARN][bootstrap] Unable to lock JVM Memory: error=12,reason=Cannot allocate memory Apr 28 10:27:29 p-es-1 elasticsearch: [2016-04-28 10:27:29,981][WARN][bootstrap] This can result in part of the JVM being swapped out. Apr 28 10:27:29 p-es-1 elasticsearch: [2016-04-28 10:27:29,981][WARN][bootstrap] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536 Apr 28 10:27:29 p-es-1 elasticsearch: [2016-04-28 10:27:29,981][WARN][bootstrap] These can be adjusted by modifying /etc/security/limits.conf, for example: Apr 28 10:27:29 p-es-1 elasticsearch: # allow user 'elasticsearch' mlockall Apr 28 10:27:29 p-es-1 elasticsearch: elasticsearch soft memlock unlimited Apr 28 10:27:29 p-es-1 elasticsearch: elasticsearch hard memlock unlimited Apr 28 10:27:29 p-es-1 elasticsearch: [2016-04-28 10:27:29,981][WARN][bootstrap] If you are logged in interactively, you will have to re-login for the new limits to take effect. Apr 28 10:27:30 p-es-1 elasticsearch: [2016-04-28 10:27:30,184][INFO][node] [p-es-1] version[2.3.1], pid[3126], build[bd98092/2016-04-04T12:25:05Z] Apr 28 10:27:30 p-es-1 elasticsearch: [2016-04-28 10:27:30,184][INFO][node] [p-es-1] initializing ... Apr 28 10:27:30 p-es-1 elasticsearch: Exception in thread "main" java.lang.IllegalArgumentException: Plugin [license] is incompatible with Elasticsearch [2.3.1]. Was designed for version [2.1.1] Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:118) Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:378) Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.plugins.PluginsService.(PluginsService.java:128) Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.node.Node.(Node.java:158) Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.node.Node.(Node.java:140) Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143) Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270) Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35) Apr 28 10:27:30 p-es-1 elasticsearch: Refer to the log for complete error details. Apr 28 10:27:30 p-es-1 systemd: elasticsearch.service: main process exited, code=exited, status=1/FAILURE Apr 28 10:27:30 p-es-1 systemd: Unit elasticsearch.service entered failed state. Apr 28 10:27:30 p-es-1 systemd: elasticsearch.service failed. ``` Looks like a lot of Java issues. Anyone else have any thoughts? ---
**Author:** ![Bombe](https://avatars.discourse-cdn.com/v4/letter/b/43a26b/32.png) [@Bombe](https://discuss.elastic.co/u/Bombe)\ **Post date:** [April 28, 2016, 5:36pm UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-and-cluster-is-down/48716/2 "2016-04-28T17:36:16Z")
> [@jnpetty](#): > > Plugin [license] is incompatible with Elasticsearch [2.3.1]. Was designed for version [2.1.1] I think this line speaks for itself: update the license plugin to the version matching your node and you should be good to go. 🙂 ---
**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\ **Post date:** [April 28, 2016, 5:39pm UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-and-cluster-is-down/48716/3 "2016-04-28T17:39:28Z")
Anyway to figure out what plugin? I think we only have Marvel ---
**Author:** ![Bombe](https://avatars.discourse-cdn.com/v4/letter/b/43a26b/32.png) [@Bombe](https://discuss.elastic.co/u/Bombe)\ **Post date:** [April 28, 2016, 5:41pm UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-and-cluster-is-down/48716/4 "2016-04-28T17:41:00Z")
For marvel you also need the “license” plugin, which coincidentally is the plugin complaining about the version mismatch. ---
**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\ **Post date:** [April 28, 2016, 6:10pm UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-and-cluster-is-down/48716/5 "2016-04-28T18:10:17Z")
Sorry im rather new to ELK, show do I go about updating plugins? You would think they would get upgraded with ES. ---
**Author:** ![Bombe](https://avatars.discourse-cdn.com/v4/letter/b/43a26b/32.png) [@Bombe](https://discuss.elastic.co/u/Bombe)\ **Post date:** [April 28, 2016, 6:15pm UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-and-cluster-is-down/48716/6 "2016-04-28T18:15:09Z")
That process is detailed at [elastic.co](https://www.elastic.co/guide/en/marvel/current/installing-marvel.html#upgrading). Basically, 1. Disable shard allocation. 2. Stop Elasticsearch. 3. `bin/plugin remove marvel-agent` (replace `marvel-agent` with the name of the plugin you’re updating, of course) 4. `bin/plugin install marvel-agent` 5. Restart Elasticsearch. 6. Reenable shard allocation. Repeat steps 2 thru 5 for every node before proceeding to 6. ---
**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\ **Post date:** [April 28, 2016, 8:38pm UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-and-cluster-is-down/48716/7 "2016-04-28T20:38:44Z")
Alright I was able to get the services up, but the cluster is still "Red". Looks like all of my shards are unassigned: ``` { "cluster_name" : "Elasticsearch-Cluster-1", "status" : "red", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 101, "active_shards" : 101, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 109, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 48.095238095238095 ``` } I havent been able to find an easy way of reassigning the shards. Any thoughts on the best way to do this? ---
**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\ **Post date:** [April 28, 2016, 8:41pm UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-and-cluster-is-down/48716/8 "2016-04-28T20:41:30Z")
I should note, that I had to hard shutdown the node servers due to a environmental issue in our datacenter. So now im trying to recover. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:55pm UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-and-cluster-is-down/48716/9 "2017-07-05T22:55:24Z")