Elasticsearch: Version 6.5.4
Logstash: Version 6.5.4
OS: CentOS 6
elasticsearch.yml:
node.name: dev
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 127.0.0.1
http.port: 9200
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300"]
logstash.yml:
path.data: /var/lib/logstash
path.logs: /var/log/logstash
logstash config (elasticsearch part in output section):
elasticsearch {
hosts => ["127.0.0.1:9200"]
template => "/etc/logstash/templates/bro.mapping.json"
template_name => "filebeats_bro"
index => "filebeats-%{+YYYY.MM.dd}"
template_overwrite => true
}
Reproduce:
Everything goes fine when in v6.2.3. After upgrading to v6.5.4, we rebuilt the machine, upgraded the machine hardwares. We kept all the configurations and settings the same as before. But we find we're always having that "SERVICE_UNAVAILABLE/1/state_not_recovered" exception in ES log. It seems that .kibana cannot perform search to the indices in ES. And shortly there will be "no shards available" in es log. After about 10 mins, all the indices in elasticsearch are recovered. But Logstash can not update the mapping of index. Logstash can not access to elasticsearch to get the template and keep complaining '400' error. We didn't touch any template, all the templates work fine in v6.2.3.
I tried curl -X PUT "127.0.0.1:9200/_template/{template_name}" -H "Content-Type: application/json" -d '{template}', and it works. But the index mapping still can not be updated, logstash can not get the updated template.
ES log:
[2019-02-19T20:37:45,435][INFO ][o.e.p.PluginsService ] [dev] loaded module [x-pack-monitoring]
[2019-02-19T20:37:45,435][INFO ][o.e.p.PluginsService ] [dev] no plugins loaded
[2019-02-19T20:37:48,641][INFO ][o.e.x.s.a.s.FileRolesStore] [dev] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2019-02-19T20:37:49,006][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [dev] [controller/1018] [Main.cc@109] controller (64 bit): Version 6.5.4 (Build b616085ef32393) Copyright (c) 2018 Elasticsearch BV
[2019-02-19T20:37:49,331][DEBUG][o.e.a.ActionModule ] [dev] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-02-19T20:37:50,603][INFO ][o.e.d.DiscoveryModule ] [dev] using discovery type [zen] and host providers [settings]
[2019-02-19T20:37:51,250][INFO ][o.e.n.Node ] [dev] initialized
[2019-02-19T20:37:51,251][INFO ][o.e.n.Node ] [dev] starting ...
[2019-02-19T20:37:51,341][INFO ][o.e.t.TransportService ] [dev] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2019-02-19T20:37:54,536][INFO ][o.e.c.s.MasterService ] [dev] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {dev}{M1R5ZuVfS2m5o2sqXGjoBA}{h6QfheUQTiCe_zPA5bYQCA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=33455321088, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2019-02-19T20:37:54,540][INFO ][o.e.c.s.ClusterApplierService] [dev] new_master {dev}{M1R5ZuVfS2m5o2sqXGjoBA}{h6QfheUQTiCe_zPA5bYQCA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=33455321088, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {dev}{M1R5ZuVfS2m5o2sqXGjoBA}{h6QfheUQTiCe_zPA5bYQCA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=33455321088, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2019-02-19T20:37:54,563][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [dev] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2019-02-19T20:37:54,563][INFO ][o.e.n.Node ] [dev] started
[2019-02-19T20:37:56,380][WARN ][r.suppressed ] [dev] path: /.kibana/_search, params: {ignore_unavailable=true, index=.kibana, filter_path=aggregations.types.buckets}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:166) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:152) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:297) ~[elasticsearch-6.5.4.jar:6.5.4]
at
Logstash log:
[2019-02-19T20:42:05,161][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://127.0.0.1:9200/_template/elastiflow'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client.rb:348:in `template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/http_client.rb:86:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/template_manager.rb:21:in `install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/template_manager.rb:9:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.4-java/lib/logstash/outputs/elasticsearch/common.rb:127:in `install_template'",