Logstash-filter-elasticsearch URI parsing error when ssl set to true


(Zulfiqar Haider) #1

This is my plugin config:

   elasticsearch {
        hosts => [ "${LS_ES_HOST1}" ]
        user     => "${LS_ES_USER}"
        password => "${LS_ES_PASSWORD}"
        ssl      => true
        ca_file   => "${LS_ES_CACERTS}"
        index => "cmx_customers"
        query => "cmx_cdir:%{[cdir]}"
        fields => { "customer_name" => "cmx_customer_name" }
    }

When I try to use this filter to run a query against an ES cluster that has SSL enabled, I get the following errors when the plugin is loaded:

2019-01-10T16:26:13,542][INFO ][logstash.filters.elasticsearch] New ElasticSearch filter client {:hosts=>[{:host=>"9.42.83.106:9200", :scheme=>"https"}]}
[2019-01-10T16:26:13,543][INFO ][logstash.filters.elasticsearch] New ElasticSearch filter client {:hosts=>[{:host=>"9.42.83.106:9200", :scheme=>"https"}]}
...
...
[2019-01-10T16:26:13,753][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"cmx_customers", :query=>"cmx_cdir:CFO", :event=>#LogStash::Event:0x4504eb02, :error=>"#<URI::InvalidURIError: bad URI(is not URI?): https://{:host=>"9.42.83.106:9200", :scheme=>"https"}:https>"}

As seen above it's enclosing the hostname within quotes ("), and is probably barfing on that. This does not happen when ssl => false. Interestingly, on each subsequent run, the hostname=> gets recursively appended to the error message resulting in something like this:

[2019-01-10T16:26:15,559][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"cmx_customers", :query=>"cmx_cdir:CC6", :event=>#LogStash::Event:0x39987564, :error=>"#<URI::InvalidURIError: bad URI(is not URI?): https://{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>
...
...
{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>"9.42.83.106:9200", :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>0}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>0}, :scheme=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>0}, :scheme=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :protocol=>"https", :port=>0}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https"}, :scheme=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>0},
...
...
:scheme=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>0}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>0}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>0}, :scheme=>"https"}, :scheme=>"https"}:https>"}

I am new to the forums and searched this issue but I am surprised no one has ever run into this? I am using version 6.5 of everything.

Would appreciate any help or pointers on how to resolve this.


#2

Specify all hosts with their https:// prefix and do not specify the ssl attribute.