Hi,
Not sure if this should be in LS or ES, but thought I'd try here first. Up until recently, I was running ES 2.4 and LS 5.6 and all was well. Recently updated ES to 5.6 and now LS only ever gets a 400 bad request response from ES. This happens when it tries to install a template and when running a health check.
Initially, I thought this was related to the templates I'm using, so using the console worked out what was wrong with them and fixed them (I had to do it this way as there was nothing more useful than the 400 response code). I can now insert a template into the console and there are no errors or warnings. However, when LS starts, it still gets a 400 error. I've tried removing the template and restarting LS, but the same result, only this time, the template isn't in ES.
I've tried searching for similar issues, but can't find any that match what I'm getting. I've also tried searching through the docs, but there's very little regarding upgrades and elastic cloud and there's nothing related to this in the logs in the cloud portal/console. I'm at a loss as to what is causing these problems. Does anyone know where I can get more info or what could be causing these 400 responses?
Example of the errors:
[WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash:xxxxxx@<cluster-id>.eu-west-1.aws.found.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '400' contacting Elasticsearch at URL 'https://<cluster-id>.eu-west-1.aws.found.io:9243/'"}
...
[2018-10-17T12:22:54,307][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/templates/elasticsearch-call-data-template.json"}
[2018-10-17T12:22:54,310][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-index-*", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "index"=>true, "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "index"=>true, "norms"=>false, "fields"=>{"raw"=>{"type"=>"keyword", "index"=>true, "ignore_above"=>256}}}}}], "properties"=>{"@version"=>{"type"=>"keyword", "index"=>true}, "geoip"=>{"type"=>"object", "dynamic"=>true, "properties"=>{"location"=>{"type"=>"geo_point"}}}}}}}}
[2018-10-17T12:22:54,390][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'https://<cluster-id>.eu-west-1.aws.found.io:9243/_template/logstash-index'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.3-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:279:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:266:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:361:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:265:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:273:in `head'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:331:in `template_exists?'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:78:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.3-java/lib/logstash/outputs/elasticsearch/template_manager.rb:27:in `install'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.3-java/lib/logstash/outputs/elasticsearch/template_manager.rb:9:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.3-java/lib/logstash/outputs/elasticsearch/common.rb:58:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.3-java/lib/logstash/outputs/elasticsearch/common.rb:25:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in `register_plugins'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:310:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:235:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:408:in `start_pipeline'"]}
Example of an elasticsearch output in the logstash config:
output {
if [type] == "calldata" {
elasticsearch {
hosts => ["<clusterid>.eu-west-1.aws.found.io:9243"]
ssl => true
user => "logstash-user"
password => "<pass>"
template => "/etc/logstash/templates/elasticsearch-call-data-template.json"
template_name => "logstash-index"
index => "logstash-index-%{+YYYY.ww}"
}
}
}
Any help would be appreciated. Cluster ID=ea33a6
Cheers,
Darren