Hi all,
I am having trouble getting logstash to only do something, when a change in an index is detected. I have an index inputs, where the docs needs to be split out in separate files - but only if the doc is not already a file.
So I have made an extra index called inputs_list, where every doc contains the name of a file made. Now I want to - in logstash - loop through all of the docs in inputs, and make a file only, if the value of the name-field of the doc also appears in the index inputs_list.
For this I am using the elastic filter plugin, but I can't seem to get it to work properly.
My logstash code is as follows:
input {
elasticsearch {
hosts => ["https://USER:PASS@HOSTNAME:PORT/"]
index => "inputs"
query => '{"query": { "match_all": {} } }' scroll => "5m"
add_field => {"new_id" => "name"}
}
}
filter {
elasticsearch {
hosts => ["https://HOSTNAME:PORT/"]
ssl => true
user => "logstash_internal"
password => "xxxxxxx"
index => "inputs_liste"
query => "name:%{[new_id]}"
fields => {
"name" => "[name]"
}
}
}
I have used various tutorials and questions/answers here from the community, and it seemed to help, that I split up username and password and set ssl up in the filter. But it now gives me the following error:
Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Manticore::ResolutionFailure: No such host is known (https)>, :backtrace=>["C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in `block in initialize'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in `call'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:274:in `call_once'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:158:in `code'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `block in perform_request'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in `perform_request'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/ping.rb:20:in `ping'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.6.0/lib/logstash/filters/elasticsearch.rb:192:in `test_connection!'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.6.0/lib/logstash/filters/elasticsearch.rb:74:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:56:in `register'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:191:in `block in register_plugins'", "org/jruby/RubyArray.java:1792:in `each'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:190:in `register_plugins'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:446:in `maybe_setup_out_plugins'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:203:in `start_workers'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:145:in `run'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:104:in `block in start'"], :thread=>"#<Thread:0x68c73907 run>"}
If I remove the "https://" from the hotsname in the filter section, this error appear:
Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Elasticsearch::Transport::Transport::Errors::BadRequest: [400] >, :backtrace=>["C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:202:in `__raise_transport_error'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:319:in `perform_request'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/ping.rb:20:in `ping'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.6.0/lib/logstash/filters/elasticsearch.rb:192:in `test_connection!'", "C:/Elastic/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.6.0/lib/logstash/filters/elasticsearch.rb:74:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:56:in `register'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:191:in `block in register_plugins'", "org/jruby/RubyArray.java:1792:in `each'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:190:in `register_plugins'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:446:in `maybe_setup_out_plugins'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:203:in `start_workers'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:145:in `run'", "C:/Elastic/logstash-7.0.0/logstash-core/lib/logstash/java_pipeline.rb:104:in `block in start'"], :thread=>"#<Thread:0x113e5fb run>"}
What do I do from here? Is my syntax off?