elaair
(Will)
August 4, 2020, 11:29pm
1
I am using logstash 7.8 and am seeing this error while using the elasticsearch filter:
:error=>"Illegal character in authority at index 8: https://{:host=>\"fqdn.here.com\", :scheme=>\"https\", :protocol=>\"https\", :port=>9200}:9200/data-activedirectory-2020
snip
what does this mean?
The config part
elasticsearch {
hosts => ['FQDN.com']
ssl => true
ca_file => "/opt/logstash/certs/FILE.crt"
user => "username"
password => "pASS"
index => "data-activedirectory-2020"
query => "sAMAccountName:%{[user]}"
fields => { "title" => "title" "department" => "department" "division" => "division" }
}
Badger
August 4, 2020, 11:43pm
2
elaair:
:error=>"Illegal character in authority at index 8: https://{:host=>\"fqdn.here.com\", :scheme=>\"https\", :protocol=>\"https\", :port=>9200}:9200/data-activedirectory-2020
Whilst building the URI to use in the query it has used
{:host=>\"fqdn.here.com\", :scheme=>\"https\", :protocol=>\"https\", :port=>9200}
as the hostname, resulting in an invalid character (the open brace "{") after the first 8 characters of the URI (https://).
That suggests that the value of the hosts option is not what you think it is.
elaair
(Will)
August 5, 2020, 12:17am
3
I am not seeing any issues or missing braces anywhere. I will restart in debug after tonights file processing completes.
elaair
(Will)
August 5, 2020, 1:21am
4
odd with the repeated hosts
[2020-08-05T01:05:48,375][INFO ][logstash.filters.elasticsearch] New ElasticSearch filter client {:hosts=>[{:host=>{:host=>"usbbcssa0001.corp.inbaxalta.com", :scheme=>"https", :protocol=>"https", :port=>9200}, :scheme=>"https"}]}
Jenni
August 5, 2020, 1:31am
5
I think you've run into an issue that seems to have been open for a while…
Specify all hosts with their https:// prefix and do not specify the ssl attribute.
When performing an Elasticsearch filter query, I get the below error (I've replaced sensitive values with an equivilent number of Xs.
[WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"servicenow-*", :error=>"Illegal character in authority at index 8: https://{:host=>\"xxxxxxxxx.xxxxxxxxx.net:9200\", :scheme=>\"https\", :protocol=>\"https\", :port=>9200}:9200/servicenow-%2A/_search?q=number%3AXXX0029703&size=1&sort=%40timestamp%3Adesc"}
What does…
opened 09:14AM - 16 Apr 20 UTC
closed 01:54PM - 12 Aug 21 UTC
I' trying to use this filter plugin. When i run the it with logstash 7.6.2 and n… ewest filter version 3.7.1, I receive the following error:
`[WARN ] 2020-04-16 09:05:18.073 [[main]>worker1] elasticsearch - Failed to query elasticsearch for previous event {:index=>"proxy-blacklist", :error=>"Illegal character in authority at index 8: https://{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>{:host=>\"elastic01.example.org\", :scheme=>\"https\", :protocol=>\"https\", :port=>9200}, :scheme=>\"https\", :protocol=>\"https\", :port=>9200}, :scheme=>\"https\", :protocol=>\"https\", :port=>9200}, :scheme=>\"https\", :protocol=>\"https\", :port=>9200}, :scheme=>\"https\"}, :scheme=>\"https\"}, :scheme=>\"https\"}, :scheme=>\"https\", :protocol=>\"https\", :port=>9200}, :scheme=>\"https\"}, :scheme=>\"https\"}, :scheme=>\"https\"}, :scheme=>\"https\", :protocol=>\"https\", :port=>9200}, :scheme=>\"https\"}, :scheme=>\"https\"}, :scheme=>\"https\", :protocol=>\"https\", :port=>9200}, :scheme=>\"https\", :protocol=>\"https\", :port=>9200}, :scheme=>\"https\"}, :scheme=>\"https\"}:9200/proxy-blacklist/_search?q=domain.keyword%3A%2F%28www.%29%3Fchat.example.org%2F&size=1&sort=%40timestamp%3Adesc"}`
So it comes down to:
`Illegal character in authority at index 8`
I've found many users which seem to have this problem but no solution at all. Some solve this by not using "ssl => true" and using the prefix "https://" in the hosts section instead. This does not solve the error for me. My filter config looks like this:
> filter {
> grok {
> patterns_dir => [ "/etc/logstash/patterns/patterns" ]
> match => [ "message", "%{SQUIDACCESS}" ]
> }
> elasticsearch {
> hosts => [ "elastic.example.org" ]
> user => "elastic"
> password => "changeme"
> ssl => true
> ca_file => "/etc/logstash/ca.crt"
> index => "blacklist"
> query => "domain.keyword:/(www.)?%{zone}/"
> fields => { "domid" => "domid" }
> fields => { "source" => "source" }
> }
> }
elaair
(Will)
August 5, 2020, 2:48am
6
I did run across that and indeed one of my config from something I was testing in Febuary has the ssl => true commented out and the https in the hosts section. I started with that type of config but when I use it logstash will not start.
Jenni
August 5, 2020, 2:52am
7
What error message do you get?
elaair
(Will)
August 5, 2020, 3:00am
8
[2020-08-05T02:59:35,888][ERROR][logstash.javapipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Manticore::SocketException: Connection refused (Connection refused)>, :backtrace=>["/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in `block in initialize'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in `call'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:274:in `call_once'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:158:in `code'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `block in perform_request'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in `perform_request'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/ping.rb:20:in `ping'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.7.1/lib/logstash/filters/elasticsearch.rb:270:in `test_connection!'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.7.1/lib/logstash/filters/elasticsearch.rb:92:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:216:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:215:in `register_plugins'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:520:in `maybe_setup_out_plugins'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:228:in `start_workers'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:170:in `run'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:125:in `block in start'"], "pipeline.sources"=>["/opt/logstash/MetricsResilientandVaronis/conf.d/resilient.conf"], :thread=>"#<Thread:0x5f30b9c run>"}
[2020-08-05T02:59:35,908][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2020-08-05T02:59:36,099][INFO ][logstash.runner ] Logstash shut down.
elaair
(Will)
August 5, 2020, 3:00am
9
I missed one line.
[2020-08-05T02:59:35,786][INFO ][logstash.filters.elasticsearch] New ElasticSearch filter client {:hosts=>["https://usbbcssa0001.corp.inbaxalta.com"]}
[2020-08-05T02:59:35,888][ERROR][logstash.javapipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Manticore::SocketException: Connection refused (Connection refused)>, :backtrace=>["/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in `block in initialize'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in `call'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:274:in `call_once'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:158:in `code'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `block in perform_request'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in `perform_request'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/ping.rb:20:in `ping'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.7.1/lib/logstash/filters/elasticsearch.rb:270:in `test_connection!'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.7.1/lib/logstash/filters/elasticsearch.rb:92:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:216:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:215:in `register_plugins'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:520:in `maybe_setup_out_plugins'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:228:in `start_workers'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:170:in `run'", "/opt/logstash/MetricsResilientandVaronis/logstash-7.8.0/logstash-core/lib/logstash/java_pipeline.rb:125:in `block in start'"], "pipeline.sources"=>["/opt/logstash/MetricsResilientandVaronis/conf.d/resilient.conf"], :thread=>"#<Thread:0x5f30b9c run>"}
[2020-08-05T02:59:35,908][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2020-08-05T02:59:36,099][INFO ][logstash.runner ] Logstash shut down.
Jenni
August 5, 2020, 7:24am
10
But you have made sure that you can connect to ES from this server (e.g. with Curl), that it's definitely only Logstash that has the connection issues?
(You can just edit your response if you have to correct something ;))
elaair
(Will)
August 5, 2020, 12:58pm
11
Yes, The same config that I am doing the lookup from is also writing the event to ES. I have also tested with CURL.
Jenni
August 5, 2020, 1:38pm
12
Hm. Unfortunately no other workaround comes to my mind. Hopefully someone else knows more.
elaair
(Will)
August 5, 2020, 6:37pm
13
at some point I had removed the :9200 from the hosts line. When I put that back logstash starts and actually runs the lookup.
system
(system)
Closed
September 2, 2020, 6:37pm
14
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.