Unable to start Elasticsearch and Cluster is down

Elastic Stack Elasticsearch
1

I have a three node cluster and one of the nodes in the cluster is failing to start the ES services. Due to this one node not starting the whole cluster is no longer running. When attempting to start this service I am seeing the following errors in /var/log/messages:

Apr 28 10:27:29 p-es-1 systemd: Starting Elasticsearch...
Apr 28 10:27:29 p-es-1 systemd: Started Elasticsearch.
Apr 28 10:27:29 p-es-1 elasticsearch: [2016-04-28 10:27:29,980][WARN ][bootstrap                ] Unable to lock JVM Memory: error=12,reason=Cannot allocate memory
Apr 28 10:27:29 p-es-1 elasticsearch: [2016-04-28 10:27:29,981][WARN ][bootstrap                ] This can result in part of the JVM being swapped out.
Apr 28 10:27:29 p-es-1 elasticsearch: [2016-04-28 10:27:29,981][WARN ][bootstrap                ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
Apr 28 10:27:29 p-es-1 elasticsearch: [2016-04-28 10:27:29,981][WARN ][bootstrap                ] These can be adjusted by modifying /etc/security/limits.conf, for example:
Apr 28 10:27:29 p-es-1 elasticsearch: # allow user 'elasticsearch' mlockall
Apr 28 10:27:29 p-es-1 elasticsearch: elasticsearch soft memlock unlimited
Apr 28 10:27:29 p-es-1 elasticsearch: elasticsearch hard memlock unlimited
Apr 28 10:27:29 p-es-1 elasticsearch: [2016-04-28 10:27:29,981][WARN ][bootstrap                ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
Apr 28 10:27:30 p-es-1 elasticsearch: [2016-04-28 10:27:30,184][INFO ][node                     ] [p-es-1] version[2.3.1], pid[3126], build[bd98092/2016-04-04T12:25:05Z]
Apr 28 10:27:30 p-es-1 elasticsearch: [2016-04-28 10:27:30,184][INFO ][node                     ] [p-es-1] initializing ...
Apr 28 10:27:30 p-es-1 elasticsearch: Exception in thread "main" java.lang.IllegalArgumentException: Plugin [license] is incompatible with Elasticsearch [2.3.1]. Was designed for version [2.1.1]
Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:118)
Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:378)
Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:128)
Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.node.Node.<init>(Node.java:158)
Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.node.Node.<init>(Node.java:140)
Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)
Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
Apr 28 10:27:30 p-es-1 elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Apr 28 10:27:30 p-es-1 elasticsearch: Refer to the log for complete error details.
Apr 28 10:27:30 p-es-1 systemd: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Apr 28 10:27:30 p-es-1 systemd: Unit elasticsearch.service entered failed state.
Apr 28 10:27:30 p-es-1 systemd: elasticsearch.service failed.

Looks like a lot of Java issues. Anyone else have any thoughts?

2

I think this line speaks for itself: update the license plugin to the version matching your node and you should be good to go. :slight_smile:

3

Anyway to figure out what plugin? I think we only have Marvel

4

For marvel you also need the “license” plugin, which coincidentally is the plugin complaining about the version mismatch.

5

Sorry im rather new to ELK, show do I go about updating plugins? You would think they would get upgraded with ES.

6

That process is detailed at elastic.co. Basically,

  1. Disable shard allocation.
  2. Stop Elasticsearch.
  3. bin/plugin remove marvel-agent (replace marvel-agent with the name of the plugin you’re updating, of course)
  4. bin/plugin install marvel-agent
  5. Restart Elasticsearch.
  6. Reenable shard allocation.

Repeat steps 2 thru 5 for every node before proceeding to 6.

7

Alright I was able to get the services up, but the cluster is still "Red". Looks like all of my shards are unassigned:

{
  "cluster_name" : "Elasticsearch-Cluster-1",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 101,
  "active_shards" : 101,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 109,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 48.095238095238095

}

I havent been able to find an easy way of reassigning the shards. Any thoughts on the best way to do this?

8

I should note, that I had to hard shutdown the node servers due to a environmental issue in our datacenter. So now im trying to recover.