Logstash-filter-elasticsearch URI parsing error when ssl set to true

Elastic Stack Logstash
1

This is my plugin config:

   elasticsearch {
        hosts => [ "${LS_ES_HOST1}" ]
        user     => "${LS_ES_USER}"
        password => "${LS_ES_PASSWORD}"
        ssl      => true
        ca_file   => "${LS_ES_CACERTS}"
        index => "cmx_customers"
        query => "cmx_cdir:%{[cdir]}"
        fields => { "customer_name" => "cmx_customer_name" }
    }

When I try to use this filter to run a query against an ES cluster that has SSL enabled, I get the following errors when the plugin is loaded:

2019-01-10T16:26:13,542][INFO ][logstash.filters.elasticsearch] New ElasticSearch filter client {:hosts=>[{:host=>"9.42.83.106:9200", :scheme=>"https"}]}
[2019-01-10T16:26:13,543][INFO ][logstash.filters.elasticsearch] New ElasticSearch filter client {:hosts=>[{:host=>"9.42.83.106:9200", :scheme=>"https"}]}
...
...
[2019-01-10T16:26:13,753][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"cmx_customers", :query=>"cmx_cdir:CFO", :event=>#LogStash::Event:0x4504eb02, :error=>"#<URI::InvalidURIError: bad URI(is not URI?): https://{:host=>"9.42.83.106:9200", :scheme=>"https"}:https>"}

As seen above it's enclosing the hostname within quotes ("), and is probably barfing on that. This does not happen when ssl => false. Interestingly, on each subsequent run, the hostname=> gets recursively appended to the error message resulting in something like this:

[2019-01-10T16:26:15,559][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"cmx_customers", :query=>"cmx_cdir:CC6", :event=>#LogStash::Event:0x39987564, :error=>"#<URI::InvalidURIError: bad URI(is not URI?): https://{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>
...
...
{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>"9.42.83.106:9200", :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>0}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>0}, :scheme=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>0}, :scheme=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :protocol=>"https", :port=>0}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https"}, :scheme=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>0},
...
...
:scheme=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>0}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https"}, :scheme=>"https"}, :scheme=>"https", :protocol=>"https", :port=>0}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>"https"}, :scheme=>"https", :protocol=>"https", :port=>0}, :scheme=>"https"}, :scheme=>"https"}:https>"}

I am new to the forums and searched this issue but I am surprised no one has ever run into this? I am using version 6.5 of everything.

Would appreciate any help or pointers on how to resolve this.

2

Specify all hosts with their https:// prefix and do not specify the ssl attribute.

1 Like
3

Thanks. This works correctly in the following cases:

  • Local logstash 6.5.4 to elastic 6.5.4
  • Local logstash 6.5.4 to elastic 6.4.x (single staging node)
  • Local logstash 6.5.4 to elastic 6.4.x (production cluster with 2 data, 3 master, and 2 ingest nodes)
  • Staging logstash 6.4.3 to elastic staging 6.4.x

However, I am having a problem successfully querying the index field when running from a production logstash 6.4.3 instance to our production ES cluster. I get the following error:

[2019-01-23T17:53:10,416][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"cmx_customers", :query=>"cmx_cdir:Q21", :event=>#LogStash::Event:0x431e79f3, :error=>#Faraday::SSLError}

I know the cmx_customers index exists and the cmx_cdir field with that value exists. The filter configuration works when querying from that es instance from my local logstash (6.5.x).

Any insights?

4

Off the top of my head no. Do the debug logs give you any more detail? I did run into an issue a while back where I had to change the hostnames in the url to the actual IP addresses as it was causing certificate failures....which produce the Faraday::SSLError message

5

I am already using actual IP addresses.

I don't see any additional information in the logstash-plain.log. I am not running with the --verbose or -debug options. Is there another log file I should be looking at that contains additional debug info?
I did see somewhere that the error logging was improved after 6.4.3 so I might just update the plugin version and see if it gives me a better error message.

Interestingly, I was helping out someone else use this filter plugin and they were seeing the same issue when running on a Mac OS connecting to the same ES cluster, but when I ran the exact same config from my RHEL 7.5 it works! So the plugin has no issue running from my RHEL machine but not from the RHEL logstash collector or from that person's OSX machine.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.