Our trial license for commercial plug-ins expired a few weeks ago. All commercial plug-ins were either uninstalled or disabled at that time. When trying to re-start the cluster with a new configuration just now, we first tried using a Basic license key (obtained by request from the Web site), then a temporary trial license key obtained from Elastic (which would support premium features, if they were installed).
In neither instance, are we able to restart the cluster. Neither key seems to be accepted.
Below are some details of our setup.
Any suggestions?
Thank you
======================================================
Command used to apply the new license
curl -XPUT 'http://node01:9200/_license?master_timeout=10m' -d @[xxxxx]-non_production-0c668ec8-14da-4144-8909-08d4992bd733-v2.json
======================================================
Health Status Check
[root@node01 logs]# curl -XGET 'localhost:9200/_cluster/health?pretty'
{
"cluster_name" : "[xxxx]",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 12,
"unassigned_shards" : 556,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 3,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 196492,
"active_shards_percent_as_number" : 0.0
}
======================================================
What is in the tail of the elasticsearch.log file
[2017-03-14 23:41:08,561][INFO ][license.plugin.core ] [node01] license [a3a0c44e-5799-411d-acdf-c55a0d83fa7b] - expired
[2017-03-14 23:41:08,567][ERROR][license.plugin.core ] [node01]
LICENSE EXPIRED ON [Tuesday, February 28, 2017]. IF YOU HAVE A NEW LICENSE, PLEASE
UPDATE IT. OTHERWISE, PLEASE REACH OUT TO YOUR SUPPORT CONTACT.
[2017-03-14 23:41:09,125][INFO ][gateway ] [node01] recovered [62] indices into cluster_state
[2017-03-14 23:41:39,177][WARN ][discovery.zen.publish ] [node01] timed out waiting for all nodes to process published state [4] (timeout [30s], pending nodes: [{node02}{XAud-zghQTOHfwpi-sG5oA}{10.0.2.143}{10.0.2.143:9300}{master=false}, {node04}{wcJugRQ8T1qwpBxcsdLKgg}{10.0.2.199}{10.0.2.199:9300}{master=false}])
[2017-03-14 23:41:49,969][WARN ][rest.suppressed ] path: /_license, params: {master_timeout=10m}
ProcessClusterEventTimeoutException[failed to process cluster event (register license [0c668ec8-14da-4144-8909-08d4992bd733]) within 30s]
at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:349)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
======================================================
Pending Tasks
[root@node01 logs]# curl 'localhost:9200/_cat/pending_tasks?v'
insertOrder timeInQueue priority source
8 5.9m HIGH cluster_reroute(async_shard_fetch)
10 5.4m HIGH add_listener
11 2.3m HIGH add_listener