S3 input plugin behind proxy


#1

Has anybody gotten the s3 input plugin working behind a HTTP proxy?

All I get is the below?

{:timestamp=>"2015-10-15T15:06:03.343000+0200", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n  Plugin: <LogStash::Inputs::S3 access_key_id=>\"XXX\", secret_access_key=>\"XXX\", aws_credentials_file=>\"/etc/logstash/aws\", bucket=>\"xxx\", region=>\"eu-west-1\", type=>\"s3\", proxy_uri=>\"http://updater:xxx@192.168.0.200:3128\", debug=>true, codec=><LogStash::Codecs::Plain charset=>\"UTF-8\">, use_ssl=>true, delete=>false, interval=>60, temporary_directory=>\"/var/lib/logstash/logstash\">\n  Error: Connection refused - Connection refused", :level=>:error}

My conf looks like this:

input {
       s3 {
               access_key_id => "XXX"
               secret_access_key => "XXX"
               aws_credentials_file => "/etc/logstash/aws"
               bucket => "xxx"
               region => "eu-west-1"
               type => "s3"
               proxy_uri => "http://updater:xxx@192.168.0.200:3128"
       }
}

There are no entries in my proxy-servers logfile, indicating, that logstash is not even trying to hit it...


(Joe) #2

I have not been able to either.
I'm using 1.5.4 and I used a proxy server and can see that logstash is still not honoring the proxy_uri that I entered even though there is a github issue noting that it was supposedly closed around the time that you posted this:


(Ed) #3

I put this in my elasticsearch.yml

cloud:
    aws:
        protocol:   ${S3_PROTOCOL}
        proxy_host: ${S3_HTTPS_PROXY}
        proxy_port: ${S3_HTTPS_PROXY_PORT}
OUT=`curl -ks "http://$HOST:$PORT/_snapshot/s3_repo-$DATE" --data-binary @- --no-buffer -XPOST << _EOF_ 
{
  "type": "s3",
  "settings": {
        "access_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXX",
        "secret_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXX",
        "bucket": "abcdefg",
        "base_path": "blah-$DATE",
        "compress": true,
        "server_side_encryption": true,
        "max_retries": 10,
        "canned_acl": "private"
  }
}

works just fine for me in 2.4 and did the same for 5.2


(system) #4