Hi All,
I am using S3 plugin to ingest logs from Oracle Cloud Infrastructure bucket. They have advised that the S3 plugin is what can be used. So I have now configured the input config as:
input {
s3 {
bucket => "logs"
endpoint => "https://<object_storage_namespace>.compat.objectstorage.<region>.oraclecloud.com"
region => "uk-london-1"
access_key_id => "*************************"
secret_access_key => "******************"
proxy_uri => "http://x.x.x.x:3128"
delete => false
interval => 300 # seconds
add_field => { "service" => "oci" }
codec => "json"
}
}
However in logstash logs I am getting the following error:connect' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924:in do_start'start' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/delegate.rb:83:in method_missing'start_session' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.501/lib/seahorse/client/net_http/connection_pool.rb:92:in session_for'session' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.501/lib/seahorse/client/net_http/handler.rb:71:in transmit'call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.501/lib/seahorse/client/plugins/content_length.rb:12:in call'
I can perform a successful curl on the endpoint https://<object_strorage_namespace>.compat.objectstorage..oraclecloud.com
Not sure what this error is about. Seems like some kind of authentication issue with the api endpoint? Any help would be much appreciated. Thank you.
Badger
August 15, 2020, 8:25pm
2
I suggest that, purely for debugging purposes , that you use the additional_settings option on the input to disable ssl_verify_peer. If that works then delete it and add one of
:ssl_ca_bundle
:ssl_ca_directory
:ssl_ca_store
to pass to the input the CA of the certificate the oraclecloud.com endpoint is using.
Hi Badger,
Thank you.
For the logstash s3 plugin input I do not see a parameter to disable ssl_verify_peer.
Badger
August 15, 2020, 8:43pm
4
Use the additonal_settings option. This can be used to pass many options understood by the Seahorse client library. For example...
additional_options => { "ssl_verify_peer" => false }
or
additional_options => { "ssl_ca_directory" => "/path2/unbundledCAcerts/" }
Hi Badger,
I updated the config as follows:
input {
s3 {
bucket => "logs"
endpoint => "https://<object_storage_namespace>.compat.objectstorage.<region>.oraclecloud.com"
region => "uk-london-1"
access_key_id => "******************"
secret_access_key => "*************"
proxy_uri => "http://x.x.x.x:3128"
delete => false
interval => 300 # seconds
add_field => { "service" => "oci" }
codec => "json"
additional_options => { "ssl_verify_peer" => false }
}
}
And when i do a config test using command:
/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/input-oci-bucket.conf
I get the error:
[ERROR] 2020-08-15 20:50:04.975 [LogStash::Runner] s3 - Unknown setting 'additional_options' for s3
Badger
August 15, 2020, 9:21pm
6
Sorry, it is additional_settings , not additional_options.
Thank you Badger.
I made the change and still getting the same error as before unfortuantely.
Badger
August 15, 2020, 11:08pm
8
Can you try
"additional_settings" => {
"ssl_verify_peer" => false
"http_wire_trace" => true
}
and see if the additional logging provides any clues?
Hi Badger,
Thanks. Unfortunately exact same error as before in the logs.
Badger
August 16, 2020, 12:59pm
10
I expected the same error, but I also expected additional logging. Did you not see any additional logs?
Hi Badger,
These are the logs.
[2020-08-16T06:55:32,440][ERROR][logstash.javapipeline ][main][326a9d448dcf6ccc3a9b4aa774b705c4e99438831cad07ede980820129dfd759] A plugin had an unrecoverable error. Will restart this plugin.http://x.x.x.x:3128 ", enable_metric=>true, role_session_name=>"logstash", watch_for_new_files=>true, temporary_directory=>"/tmp/logstash", include_object_properties=>false, gzip_pattern=>".gz(ip)?$">connect' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924:in do_start'start' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/delegate.rb:83:in method_missing'start_session' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.501/lib/seahorse/client/net_http/connection_pool.rb:92:in session_for'session' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.501/lib/seahorse/client/net_http/handler.rb:71:in transmit'call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.501/lib/seahorse/client/plugins/content_length.rb:12:in call'call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.501/lib/aws-sdk-core/plugins/s3_request_signer.rb:23:in call'
Badger
August 16, 2020, 2:45pm
12
What version of the s3 input are you running?
bin/logstash-plugin list --verbose logstash-input-s3
logstash-input-s3 (3.5.0)
Which I believe is the latest.
Badger
August 16, 2020, 3:18pm
14
Indeed, and it includes the fixes from 3.3.3. Before that the additional_settings option was not effective. OK, I have no further suggestions on why those settings appear not to be applied.
thank you badger.
Though I was wondering if http_poller plugin would be able to work by calling the oracle cloud bucket api endpoint?
I tried the below, no errors in logstash logs and no result in Kibana
input {
http_poller {
urls => {
test1 => {
method => get
user => "*********"
password => "*********"
url => 'https://<bucket_namespace>.compat.objectstorage.uk-london-1.oraclecloud.com'
#headers => {
# Accept => "application/json"
#}
}
}
request_timeout => 60
codec => "json"
#schedule => { every => "20s"}
schedule => { cron => "* * * * * UTC"}
metadata_target => "http_poller_metadata"
add_field => { "service" => "oci" }
proxy => "http://x.x.x.x:3128"
}
}
system
September 13, 2020, 3:20pm
16
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
