I might killed my local es instance by accident. but when I restarted the es, it is red and can't do anything literally.
curl -XGET 'http://localhost:9200/_cat/shards'
.marvel-es-2015.12.20 0 p UNASSIGNED
.marvel-es-2015.12.20 0 r UNASSIGNED
.watch_history-2015.12.10 0 p UNASSIGNED
.watch_history-2015.12.10 0 r UNASSIGNED
.triggered_watches 0 p UNASSIGNED
.triggered_watches 0 r UNASSIGNED
.marvel-es-2015.12.21 0 p UNASSIGNED
.marvel-es-2015.12.21 0 r UNASSIGNED
catalog 4 p INITIALIZING 127.0.0.1 smoker
catalog 4 r UNASSIGNED
catalog 3 p INITIALIZING 127.0.0.1 smoker
catalog 3 r UNASSIGNED
...
.marvel-es-2015.12.19 0 p UNASSIGNED
.marvel-es-2015.12.19 0 r UNASSIGNED
.kibana 0 p UNASSIGNED
.kibana 0 r UNASSIGNED
...
.watches 0 p UNASSIGNED
.watches 0 r UNASSIGNED
...
log:
[2015-12-22 10:40:03,227][INFO ][http ] [smoker] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2015-12-22 10:40:03,227][INFO ][node ] [smoker] started
[2015-12-22 10:40:08,421][INFO ][license.plugin.core ] [smoker] license [f02c3aef-0744-455d-8e06-bff9bc9e9689] - valid
[2015-12-22 10:40:08,423][ERROR][license.plugin.core ] [smoker]
License will expire on [Sunday, January 03, 2016]. If you have a new license, please update it.
Otherwise, please reach out to your support contact.
Commercial plugins operate with reduced functionality on license expiration:
- marvel
- The agent will stop collecting cluster and indices metrics
- watcher
- PUT / GET watch APIs are disabled, DELETE watch API continues to work
- Watches execute and write to the history
- The actions of the watches don't execute
[2015-12-22 10:40:08,546][INFO ][gateway ] [smoker] recovered [39] indices into cluster_state
[2015-12-22 10:40:08,827][WARN ][index.translog ] [smoker] [catalog][3] failed to delete temp file /usr/local/var/elasticsearch/ada_elasticsearch/nodes/0/indices/catalog/3/translog/translog-6253591877225990936.tlog
java.nio.file.NoSuchFileException: /usr/local/var/elasticsearch/ada_elasticsearch/nodes/0/indices/catalog/3/translog/translog-6253591877225990936.tlog ....
[2015-12-22 10:40:10,125][DEBUG][action.admin.indices.stats] [smoker] [indices:monitor/stats] failed to execute operation for shard [[catalog][4], node[uHU0-PXXTVCXjnbwSgbDbQ], [P], v[3], s[INITIALIZING], a[id=1AjK2l2cTp6w7xQp2-9_6Q], unassigned_info[[reason=CLUSTER_RECOVERED], at[2015-12-22T18:40:03.270Z]]]
[catalog][[catalog][4]] BroadcastShardOperationFailedException[operation indices:monitor/stats failed]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];
and a bunch of :
[0]: index [.marvel-es-2015.12.22], type [indices_stats], id [AVHK_dAN6t_cz8ROgyqD], message [UnavailableShardsException[[.marvel-es-2015.12.22][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@5cda97cf]]
2015-12-22 10:48:11,172][ERROR][marvel.agent ] [smoker] background thread had an uncaught exception
ElasticsearchException[failed to flush exporter bulks]
at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:104)
at org.elasticsearch.marvel.agent.exporter.ExportBulk.close(ExportBulk.java:53)
at org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:201)
at java.lang.Thread.run(Thread.java:745)
Suppressed: ElasticsearchException[failed to flush [default_local] exporter bulk]; nested: ElasticsearchException[failure in bulk execution:
...
curl -XGET 'http://localhost:9200/'
{
"name" : "smoker",
"cluster_name" : "ada_elasticsearch",
"version" : {
"number" : "2.1.0",
"build_hash" : "72cd1f1a3eee09505e036106146dc1949dc5dc87",
"build_timestamp" : "2015-11-18T22:40:03Z",
"build_snapshot" : false,
"lucene_version" : "5.3.1"
},
I tried to delete the index but 30s timeout.want to know what caused this and how to fix bring the cluster back?