Logstash - Startup - Template, Exception in start

Elastic Stack Logstash
1

ES Version 6.2.1
Logstash Version (6.0.1 and 6.2.4).

I'm observing multiple logstash templates being created, and was not clear on these.

Environment:
2 Clusters

  • Cluster 1 Core Logging Cluster for keeping business data
  • Cluster 2 Monitoring Cluster where Logstash (Xpack) monitoring, and CLuster 1 (XPack) monitoring send data

I observe below templates in these clusters

Core Logging Cluster
logstash-index-template [.logstash]

Monitoring Cluster
.monitoring-logstash [.monitoring-logstash-6-]
.monitoring-logstash-2 [.monitoring-logstash-2]

Clarifications

  1. When these templates are created? I believe .monitoring-logstash are created when xpack is enabled in logstash, but not sure why we get two versions .monitoring-logstash being created. The indices follow only .monitoring-logstash-6-* in monitoring cluster

  2. logstash-index-template? How and when it is created. I don't see any indices as specified by template

  3. Whenever I start the logstash, I see below error in it. Refer to highlighted text.
    [2018-05-04T22:03:24,209][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
    [2018-05-04T22:03:24,261][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '403' contacting Elasticsearch at URL 'https://<>:9200/_template/logstash'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:inperform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:inwith_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:inblock in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:338:in template_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:81:intemplate_install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:21:in install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:9:ininstall_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:96:in install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:26:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:in register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:42:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:in register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:353:inblock in register_plugins'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:353:inregister_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:730:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:363:instart_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290:in run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:250:inblock in start'"]}

Based on the error, I have few more queries

3.1. Logstash start is trying to search a template named logstash in Logging Cluster, which will not be found. Not sure what is being done and why I get this error. Moreover why would logstash try to install/search template in logging cluster. The output plugin has output configured to write against specific indices in pipeline at logging cluster, and access the logging cluster using specific user/pwd. The pipeline work fine though

3.2 Refer to 403 response code. What security credentials it is using to access logging cluster during startup?

Thanks for clarification

2

the elasticsearch output in your logging pipeline appears to be configured to push a template for the Elasticsearch index, but when it does so it is receiving an 403 Forbidden; is the target logging cluster password-protected? if so, you will need to set the output's user and password parameters with appropriate credentials.

3

Hi Ry

Validated the pipelines, but they all write data to indices and doesn't create a template. This is the only output configuration we have for all pipelines.

output {
elasticsearch {
id => "elasticsearch.output"
hosts => [""]
index => "%{type}.log.%{tier}"
user => ""
password => ""
ssl => true
ssl_certificate_verification => true
cacert => ""
}
}

type and tier are set at input to qualify a indice to where the data is logged to. All the output by type and tier is working fine and data is indexed. The error comes at startup, and not continuous.

Also, it would be helpful if you can provide clarification on other two queries (1 &2) around indice templates being created.

4

Documentation for template management can be found here:

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-manage_template

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.