Logstash Elasticsearch filter Bad URI Exception

Elastic Stack Logstash
1

Hello Everyone.
I have an exception in Logstash with Elasticsearch filter in the hosts parameter.
When i give manually the hosts parameter like this, all thing is good, i don't get error :

filter {
	   elasticsearch {
	      hosts                           => ["https://server_ip:9201", "https://server_ip:9202", "https://server_ip:9203"]
	      user                            => "${USERNAME}"
	      password                        => "${PASSWORD}"
	      ca_file                         => "${CERT_FILE}"
	      query 						  => "field1:%{[field1]}"
	      index                           => "index01"
	      result_size					  => 1
	      fields 						  => { 
	      										"some_field" => "some_field"
	   							             }
	      }
}

But when i am getting the hosts parameter from the .env file like this, i got a bad URI exception :

filter {
	   elasticsearch {
	      hosts                           => "${HOSTS}"
	      user                            => "${USERNAME}"
	      password                        => "${PASSWORD}"
	      ca_file                         => "${CERT_FILE}"
	      query 						  => "field1:%{[field1]}"
	      index                           => "index01"
	      result_size					  => 1
	      fields 						  => { 
	      										"some_field" => "some_field"
	   							             }
	      }
}

Here is the .env file :

HOSTS=["https://server_ip:9201", "https://server_ip:9202", "https://server_ip:9203"]
USERNAME=username
PASSWORD=password
CERT_FILE=/path/to/cert

I have tried many options for the HOSTS parameter in the .env like this without result :

  • ["https://server_ip:9201", "https://server_ip:9202", "https://server_ip:9203"]
  • ["https://server_ip:9201" "https://server_ip:9202" "https://server_ip:9203"]
  • ['https://server_ip:9201', 'https://server_ip:9202', 'https://server_ip:9203']
  • ['https://server_ip:9201' 'https://server_ip:9202' 'https://server_ip:9203']
  • "https://server_ip:9201, https://server_ip:9202, https://server_ip:9203"
  • "https://server_ip:9201 https://server_ip:9202 https://server_ip:9203"

Here is the full log from Logstash :

{"level":"ERROR","loggerName":"logstash.javapipeline","timeMillis":1673718151626,"thread":"[test]-pipeline-manager","logEvent":{"message":"Pipeline error","pipeline_id":"test","exception":{"metaClass":{"metaClass":{"metaClass":{"exception":"bad URI(is not URI?): https://192.15.10.0:9201 https://192.15.10.0:9202 https://192.15.10.0:9203","backtrace":["/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/uri/rfc3986_parser.rb:67:in `split'","/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/uri/common.rb:199:in `split'","/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/client.rb:312:in `__parse_host'","/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/client.rb:302:in `block in __extract_hosts'","org/jruby/RubyArray.java:2589:in `map'","/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/client.rb:302:in `__extract_hosts'","/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.17.1/lib/elasticsearch/transport/client.rb:154:in `initialize'","/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-7.17.1/lib/elasticsearch.rb:35:in `initialize'","/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.12.0/lib/logstash/filters/elasticsearch/client.rb:37:in `initialize'","/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.12.0/lib/logstash/filters/elasticsearch.rb:218:in `new_client'","/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.12.0/lib/logstash/filters/elasticsearch.rb:222:in `block in get_client'","/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.12.0/lib/logstash/filters/elasticsearch.rb:222:in `get_client'","/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.12.0/lib/logstash/filters/elasticsearch.rb:330:in `test_connection!'","/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.12.0/lib/logstash/filters/elasticsearch.rb:118:in `register'","org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'","/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in `block in register_plugins'","org/jruby/RubyArray.java:1821:in `each'","/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `register_plugins'","/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:599:in `maybe_setup_out_plugins'","/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:245:in `start_workers'","/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:190:in `run'","/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:142:in `block in start'"],"pipeline.sources":["/usr/share/logstash/pipeline/test/conf.d/10-test-input.conf","/usr/share/logstash/pipeline/test/conf.d/20-test-filters-01.conf","/usr/share/logstash/pipeline/test/conf.d/20-test-filters-02.conf","/usr/share/logstash/pipeline/test/conf.d/20-test-filters-03.conf","/usr/share/logstash/pipeline/test/conf.d/30-test-output.conf"],"thread":"#<Thread:0x29363338 run>"}}}}}}

Can someone help me please with this issue ?

2

Please share the full log from Logstash.

3

Check this topic

4

I update the post with the full log

5

This work for elasticsearch output, but don't work for the filter. I am using the elasticsearch filter.

6

The correct way to declare multiple hosts in an environment variable is the last one you shared, with a space between the hosts.

Something like this

ES_NODES="https://host-01:9200 https://host-02:9200 https://host-03:9200"

For the elasticsearch output you would use the hosts like this:

hosts => ["${ES_NODES}"]

I would expect that this should work as well in the Elasticsearch filter, but in your filter configuration you used like this:

hosts => "${HOSTS}"

Try to change into this to see if it will work.

hosts => ["${HOSTS}"]
8

Thank for your reply. I try but get the same error

9

That is correct. The PR that enables use of multi-valued environment variables is specific to options declared as :uri. The output uses :uri, the filter uses an array of strings.

URIs cannot contain spaces, so that can be used as a separator for multiple values, that is not true of strings.

1 Like
11

Yeah thank you, so how can i fix this issue ? Or what can i use as separator for the array string ?

12

I don't think you can. You can only pass the filter a single host using an environment variable.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.