Elastic filter plugin https error (ssl?)

Hi all,

I am having trouble getting logstash to only do something, when a change in an index is detected. I have an index inputs, where the docs needs to be split out in separate files - but only if the doc is not already a file.

So I have made an extra index called inputs_list, where every doc contains the name of a file made. Now I want to - in logstash - loop through all of the docs in inputs, and make a file only, if the value of the name-field of the doc also appears in the index inputs_list.

For this I am using the elastic filter plugin, but I can't seem to get it to work properly.

My logstash code is as follows:

input {
 elasticsearch {
     hosts => ["https://USER:PASS@HOSTNAME:PORT/"]
     index => "inputs"
     query => '{"query": { "match_all": {} } }' scroll => "5m" 
     add_field => {"new_id" => "name"}
   }
}
    filter {
    elasticsearch {
    hosts => ["https://HOSTNAME:PORT/"]
    ssl => true
    user => "logstash_internal"
    password => "xxxxxxx"
    index => "inputs_liste"
    query => "name:%{[new_id]}"
    fields => {
     "name" => "[name]"
    }
  }   
}

I have used various tutorials and questions/answers here from the community, and it seemed to help, that I split up username and password and set ssl up in the filter. But it now gives me the following error:

Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Manticore::ResolutionFailure: No such host is known (https)>, :backtrace=>["C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in `block in initialize'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in `call'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:274:in `call_once'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:158:in `code'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `block in perform_request'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in `perform_request'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/ping.rb:20:in `ping'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.6.0/lib/logstash/filters/elasticsearch.rb:192:in `test_connection!'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.6.0/lib/logstash/filters/elasticsearch.rb:74:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:56:in `register'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:191:in `block in register_plugins'", "org/jruby/RubyArray.java:1792:in `each'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:190:in `register_plugins'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:446:in `maybe_setup_out_plugins'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:203:in `start_workers'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:145:in `run'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:104:in `block in start'"], :thread=>"#<Thread:0x68c73907 run>"}

If I remove the "https://" from the hotsname in the filter section, this error appear:

 Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Elasticsearch::Transport::Transport::Errors::BadRequest: [400] >, :backtrace=>["C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:202:in `__raise_transport_error'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:319:in `perform_request'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/ping.rb:20:in `ping'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.6.0/lib/logstash/filters/elasticsearch.rb:192:in `test_connection!'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.6.0/lib/logstash/filters/elasticsearch.rb:74:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:56:in `register'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:191:in `block in register_plugins'", "org/jruby/RubyArray.java:1792:in `each'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:190:in `register_plugins'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:446:in `maybe_setup_out_plugins'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:203:in `start_workers'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:145:in `run'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:104:in `block in start'"], :thread=>"#<Thread:0x113e5fb run>"}

What do I do from here? Is my syntax off?

I stumbled upon this thread that mentioned the use of
enable_sort => false

and it works. So.. That was weird, but it works now. Hope this helps others ^^.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.