Logstash 5.1.2 connecting to elasticsearch in cloud apparantly has no template

When logstash initializes it shows no mapping template in the logs. When it goes to send data there are "can't convert nil to Array" errors. Note that this is using stunnel for https to the elastic cloud node.

Here is the error from logstash logs:

[2017-02-06T21:53:24,357][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2017-02-06T21:53:24,769][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>["http://~hidden~:~hidden~@localhost:19200"]}}
[2017-02-06T21:53:24,771][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:url=>#<URI::HTTP:0x4ab00bdd URL:http://~hidden~:~hidden~@localhost:19200>, :healthcheck_path=>"/"}
[2017-02-06T21:53:24,999][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x4ab00bdd URL:http://~hidden~:~hidden~@localhost:19200>}
[2017-02-06T21:53:24,999][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-02-06T21:53:25,060][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"undefined method []' for nil:NilClass", :class=>"NoMethodError"} [2017-02-06T21:53:25,061][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["localhost:19200"]} [2017-02-06T21:53:25,157][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250} [2017-02-06T21:53:25,159][INFO ][logstash.pipeline ] Pipeline main started [2017-02-06T21:53:25,202][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2017-02-06T21:53:58,318][ERROR][logstash.outputs.elasticsearch] An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely {:error_message=>"can't convert nil into Array", :error_class=>"TypeError", :backtrace=>["org/jruby/RubyArray.java:1462:inconcat'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:94:in join_bulk_responses'", "org/jruby/RubyArray.java:1613:ineach'", "org/jruby/RubyEnumerable.java:852:in inject'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:92:injoin_bulk_responses'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:88:in bulk'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:186:insafe_bulk'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:109:in submit'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:76:inretrying_submit'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:27:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:12:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:42:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:331:inoutput_batch'", "org/jruby/RubyHash.java:1342:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:330:inoutput_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:288:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:258:instart_workers'"]}

Fixed the "can't convert nil to Array" errors by upgrading the elasticsearch output plugin from 5.4.0 to 6.2.5. A new error now shows up for the failure to install a template:

[2017-02-06T23:42:32,073][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"undefined method []' for nil:NilClass", :class=>"NoMethodError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.5-java/lib/logstash/outputs/elasticsearch/template_manager.rb:20:inget_es_major_version'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.5-java/lib/logstash/outputs/elasticsearch/template_manager.rb:7:in install_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.5-java/lib/logstash/outputs/elasticsearch/common.rb:54:ininstall_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.5-java/lib/logstash/outputs/elasticsearch/common.rb:21:in register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:8:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:37:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:229:instart_workers'", "org/jruby/RubyArray.java:1613:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:229:instart_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:183:in run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:292:instart_pipeline'"]}

Now no errors are logged except the above failed to install template error. Even with the fix, no logs show up in kibana. Is the elasticsearch server not reporting its version properly so the correct template mappings can be found? If I read this properly, the ruby source appears to be failing in the function:

def self.get_es_major_version(client)
  get_es_version(client)["number"][0]
end

I believe I see my error. I did not follow the instructions for stunnel properly, working on using elasticsearch_http output instead of elasticsearch and using the correct configuration.

https://www.elastic.co/guide/en/cloud/current/tunneling-ssl.html

Found that logstash 5.x does support HTTPS. Had been led to believe otherwise by some of the documentation: https://www.elastic.co/support/matrix#show_compatibility

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.