Logstash 6.8 role_arn error

[Kevin_f]

Hello,

I was hoping that someone could help me figure this one out.

Basically what it is that I am trying to do in AWS is:

Account 1 (logstash instance with role 1 attached)
Account 2 (contains the s3 with role 2 created with the policies to ListBucket and GetObject)

Account role 1 is to assume Account role 2. This is to ingest logs from Account 2 S3 bucket to Account 1 logstash instance.

My logstash input.conf as shown below

input {
  s3 {
    bucket => "bucket_name"
    region => "eu-west-1"
    proxy_uri => "http://x.x.x.x:3128"
    delete => false
    interval => 300 # seconds
    prefix => "AWSLogs/"
    add_field => { "service" => "cloudtrail" }
    codec => "cloudtrail"
    role_arn => "account2_role_arn"
    tags => [  'cloudtrail', 'aws'   ]
    sincedb_path => "/srv/log/sincedb/.testsincedb"
    temporary_directory => "/srv/log/cloudtrail/temp"
  }
}

However, logstash logs keeps giving me the error:

[logstash.pipeline] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Seahorse::Client::NetworkingError: execution expired>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:119:in `initialize'", "org/jruby/RubyIO.java:1155:in `open'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:941:in `block in connect'", "org/jruby/ext/timeout/Timeout.java:99:in `timeout'", "org/jruby/ext/timeout/Timeout.java:75:in `timeout'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:939:in `connect'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924:in `do_start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:919:in `start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/delegate.rb:83:in `method_missing'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.517/lib/seahorse/client/net_http/connection_pool.rb:285:in `start_session'", 

I tested the IAM role cross account on the logstash instance using awscli commands and that works with the below where I am able to list and download the bucket contents of the Account 2 S3 onto Account 1 logstash instance itself :

aws s3 ls s3://bucket_name

and

aws s3 cp s3://bucket_name/path_to_contents ./test.json

So from the 2 awscli above results, this shows that it is nothing to do with security groups and ACLs between the 2 accounts. Nor has nothing to do with the policies set on the IAM roles.

I have done some more research and came across other posts mentioning not to specify the role_arn in the s3 input.conf file as I have done above. But doing that gives me the error:

Unable to list objects in bucket {:exception=>Aws::S3::Errors::AccessDenied, :message=>"Access Denied",

This works okay using IAM user with access_key and secret_key_id but do not want to keep rotating aws keys every 90 days for many accounts as it is tidious.

Really stuck on this one and help would be much appreciated. Thank you.

[Kevin_f]

So seems like i may have resolved the issue after all after all.

For what ever reason the s3 input plugin is not actively using the uri_proxy attribute so what I did was to add the proxy settings to the jvm.options file in /etc/logstash

-Dhttp.proxyHost=x.x.x.x
-Dhttp.proxyPort=xxxx

Restarted logstash service and all WORKED!