"SERVICE_UNAVAILABLE/1/state_not_recovered" exception in ES after upgrade to v6.5.4

Elastic Stack Elasticsearch
1

Elasticsearch: Version 6.5.4

OS: CentOS 6

elasticsearch.yml:

node.name: dev
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 127.0.0.1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9200"]

logstash.yml:

path.data: /var/lib/logstash
path.logs: /var/log/logstash

Reproduce:
Everything goes fine when in v6.2.3. After upgrading to v6.5.4, we rebuilt the machine, upgraded the machine hardwares. We kept all the configurations and settings the same as before. But we find we're always having that "SERVICE_UNAVAILABLE/1/state_not_recovered" exception in ES log. It seems that .kibana cannot perform search to the indices in ES. And shortly there will be "no shards available" in es log. After about 10 mins, all the indices in elasticsearch are recovered. But we can not update the mapping of index. Logstash can not access to elasticsearch to get the template and keep complaining '400' error. We didn't touch any template, all the templates work fine in v6.2.3.
I tried curl -X PUT "127.0.0.1:9200/_template/{template_name}" -H "Content-Type: application/json" -d '{template}', and it works. But the index mapping still can not be updated, logstash can not get the updated template.

ES log:

[2019-02-19T20:37:45,435][INFO ][o.e.p.PluginsService     ] [dev] loaded module [x-pack-monitoring]
[2019-02-19T20:37:45,435][INFO ][o.e.p.PluginsService     ] [dev] no plugins loaded
[2019-02-19T20:37:48,641][INFO ][o.e.x.s.a.s.FileRolesStore] [dev] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2019-02-19T20:37:49,006][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [dev] [controller/1018] [Main.cc@109] controller (64 bit): Version 6.5.4 (Build b616085ef32393) Copyright (c) 2018 Elasticsearch BV
[2019-02-19T20:37:49,331][DEBUG][o.e.a.ActionModule       ] [dev] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-02-19T20:37:50,603][INFO ][o.e.d.DiscoveryModule    ] [dev] using discovery type [zen] and host providers [settings]
[2019-02-19T20:37:51,250][INFO ][o.e.n.Node               ] [dev] initialized
[2019-02-19T20:37:51,251][INFO ][o.e.n.Node               ] [dev] starting ...
[2019-02-19T20:37:51,341][INFO ][o.e.t.TransportService   ] [dev] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2019-02-19T20:37:54,536][INFO ][o.e.c.s.MasterService    ] [dev] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {dev}{M1R5ZuVfS2m5o2sqXGjoBA}{h6QfheUQTiCe_zPA5bYQCA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=33455321088, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2019-02-19T20:37:54,540][INFO ][o.e.c.s.ClusterApplierService] [dev] new_master {dev}{M1R5ZuVfS2m5o2sqXGjoBA}{h6QfheUQTiCe_zPA5bYQCA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=33455321088, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {dev}{M1R5ZuVfS2m5o2sqXGjoBA}{h6QfheUQTiCe_zPA5bYQCA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=33455321088, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2019-02-19T20:37:54,563][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [dev] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2019-02-19T20:37:54,563][INFO ][o.e.n.Node               ] [dev] started
[2019-02-19T20:37:56,380][WARN ][r.suppressed             ] [dev] path: /.kibana/_search, params: {ignore_unavailable=true, index=.kibana, filter_path=aggregations.types.buckets}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:166) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:152) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:297) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.search.TransportSearchAction.lambda$doExecute$4(TransportSearchAction.java:193) ~[elasticsearch-6.5.4.jar:6.5.4]

Logstash log:

[2019-02-19T20:42:05,161][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://127.0.0.1:9200/_template/elastiflow'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client.rb:348:in `template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client.rb:86:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/template_manager.rb:21:in `install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/template_manager.rb:9:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/common.rb:127:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/common.rb:49:in `block in install_template_after_successful_connection'"]}`

Any idea?

2

Please don't post images of text as they are hardly readable and not searchable.

Instead paste the text and format it with </> icon. Check the preview window.

Not sure it's related but this is wrong:

discovery.zen.ping.unicast.hosts: ["127.0.0.1:9200"]

It should be:

discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300"]
3

Hi @dadoonet, sorry for the image, I've pasted the plain text instead.

I've tried to change it to port 9300, but the errors are still there after restarting. The ES log is still exactly the same as above. I'm quite confused now...

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.