ES Version 6.2.1
Logstash Version (6.0.1 and 6.2.4).
I'm observing multiple logstash templates being created, and was not clear on these.
Environment:
2 Clusters
- Cluster 1 Core Logging Cluster for keeping business data
- Cluster 2 Monitoring Cluster where Logstash (Xpack) monitoring, and CLuster 1 (XPack) monitoring send data
I observe below templates in these clusters
Core Logging Cluster
logstash-index-template [.logstash]
Monitoring Cluster
.monitoring-logstash [.monitoring-logstash-6-]
.monitoring-logstash-2 [.monitoring-logstash-2]
Clarifications
-
When these templates are created? I believe .monitoring-logstash are created when xpack is enabled in logstash, but not sure why we get two versions .monitoring-logstash being created. The indices follow only .monitoring-logstash-6-* in monitoring cluster
-
logstash-index-template? How and when it is created. I don't see any indices as specified by template
-
Whenever I start the logstash, I see below error in it. Refer to highlighted text.
[2018-05-04T22:03:24,209][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-05-04T22:03:24,261][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '403' contacting Elasticsearch at URL 'https://<>:9200/_template/logstash'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:inperform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:in
perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:inblock in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:in
with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:inperform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in
block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:338:intemplate_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:81:in
template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:21:ininstall'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:9:in
install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:96:ininstall_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:26:in
register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:42:in
register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:inregister_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:353:in
block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:353:in
register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:730:inmaybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:363:in
start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:250:in
block in start'"]}
Based on the error, I have few more queries
3.1. Logstash start is trying to search a template named logstash in Logging Cluster, which will not be found. Not sure what is being done and why I get this error. Moreover why would logstash try to install/search template in logging cluster. The output plugin has output configured to write against specific indices in pipeline at logging cluster, and access the logging cluster using specific user/pwd. The pipeline work fine though
3.2 Refer to 403 response code. What security credentials it is using to access logging cluster during startup?
Thanks for clarification