I have one logstash that's shipping files to s3, and one logstash that gets them from s3 and inserts them in elasticsearch. But the s3 input isn't working.
Config:
s3 {
access_key_id => "XXXX"
secret_access_key => "XXXX"
region => "eu-west-1"
bucket => "MyBucket"
delete => true
}
When running logstash -f s3.conf --debug:
{:timestamp=>"2015-10-26T15:44:17.762000+0100", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::S3 access_key_id=>\"XXX\", secret_access_key=>\"XXXX\", region=>\"eu-west-1\", bucket=>\"MyBucket\", delete=>true, debug=>true, codec=><LogStash::Codecs::P
lain charset=>\"UTF-8\">, use_ssl=>true, interval=>60, temporary_directory=>\"/tmp/logstash\">\n Error: execution expired\n Exception: Timeout::Error\n Stack: org/jruby/ext/socket/RubyTC
PSocket.java:111:in `initialize'\norg/jruby/RubyIO.java:1179:in `open'\n/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:763:in `connect'\norg/jruby/ext/timeout/Timeout.java:126:in `time
out'\n/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:763:in `connect'\n/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:756:in `do_start'\n/opt/logstash/vendor/jruby/lib/ruby/1.9/ne
t/http.rb:751:in `start'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/http/connection_pool.rb:327:in `start_session'\n/opt/logstash/vendor/bundle/jruby/1.9/gem
s/aws-sdk-v1-1.64.0/lib/aws/core/http/connection_pool.rb:127:in `session_for'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/http/net_http_handler.rb:56:in `hand
le'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:253:in `make_sync_request'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/cor
e/client.rb:289:in `retry_server_errors'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/s3/region_detection.rb:11:in `retry_server_errors'\n/opt/logstash/vendor/bundl
e/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:249:in `make_sync_request'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:509:in `client_requ
est'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:391:in `log_client_request'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/c
ore/client.rb:477:in `client_request'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:373:in `return_or_raise'\n/opt/logstash/vendor/bundle/jruby/1.9/ge
ms/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:476:in `client_request'\n(eval):3:in `list_objects'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/s3/object_collection.rb:298:in `list_request'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/s3/paginated_collection.rb:29:in `_each_item'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/collection/with_limit_and_next_token.rb:54:in `_each_batch'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/collection.rb:80:in `each_batch'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/collection.rb:47:in `each'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/s3/object_collection.rb:282:in `each'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-1.0.0/lib/logstash/inputs/s3.rb:109:in `list_new_files'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-1.0.0/lib/logstash/inputs/s3.rb:143:in `process_files'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-1.0.0/lib/logstash/inputs/s3.rb:101:in `run'\norg/jruby/RubyProc.java:271:in `call'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/interval.rb:17:in `interval'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-1.0.0/lib/logstash/inputs/s3.rb:100:in `run'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:177:in `inputworker'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:171:in `start_input'", :level=>:error, :file=>"logstash/pipeline.rb", :line=>"182", :method=>"inputworker"}
The credentials are working fine for the s3 output, and the policy is:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::MyBucket/*",
"arn:aws:s3:::MyBucket"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
Any ideas?
