S3 input, execution expired [SOLVED]


(Calle Kabo) #1

I have one logstash that's shipping files to s3, and one logstash that gets them from s3 and inserts them in elasticsearch. But the s3 input isn't working.

Config:

s3 {
  access_key_id => "XXXX"
  secret_access_key => "XXXX"
  region => "eu-west-1"
  bucket => "MyBucket"
  delete => true
}

When running logstash -f s3.conf --debug:

{:timestamp=>"2015-10-26T15:44:17.762000+0100", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n  Plugin: <LogStash::Inputs::S3 access_key_id=>\"XXX\", secret_access_key=>\"XXXX\", region=>\"eu-west-1\", bucket=>\"MyBucket\", delete=>true, debug=>true, codec=><LogStash::Codecs::P
lain charset=>\"UTF-8\">, use_ssl=>true, interval=>60, temporary_directory=>\"/tmp/logstash\">\n  Error: execution expired\n  Exception: Timeout::Error\n  Stack: org/jruby/ext/socket/RubyTC
PSocket.java:111:in `initialize'\norg/jruby/RubyIO.java:1179:in `open'\n/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:763:in `connect'\norg/jruby/ext/timeout/Timeout.java:126:in `time
out'\n/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:763:in `connect'\n/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:756:in `do_start'\n/opt/logstash/vendor/jruby/lib/ruby/1.9/ne
t/http.rb:751:in `start'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/http/connection_pool.rb:327:in `start_session'\n/opt/logstash/vendor/bundle/jruby/1.9/gem
s/aws-sdk-v1-1.64.0/lib/aws/core/http/connection_pool.rb:127:in `session_for'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/http/net_http_handler.rb:56:in `hand
le'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:253:in `make_sync_request'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/cor
e/client.rb:289:in `retry_server_errors'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/s3/region_detection.rb:11:in `retry_server_errors'\n/opt/logstash/vendor/bundl
e/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:249:in `make_sync_request'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:509:in `client_requ
est'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:391:in `log_client_request'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/c
ore/client.rb:477:in `client_request'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:373:in `return_or_raise'\n/opt/logstash/vendor/bundle/jruby/1.9/ge
ms/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:476:in `client_request'\n(eval):3:in `list_objects'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/s3/object_collection.rb:298:in `list_request'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/s3/paginated_collection.rb:29:in `_each_item'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/collection/with_limit_and_next_token.rb:54:in `_each_batch'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/collection.rb:80:in `each_batch'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/collection.rb:47:in `each'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/s3/object_collection.rb:282:in `each'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-1.0.0/lib/logstash/inputs/s3.rb:109:in `list_new_files'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-1.0.0/lib/logstash/inputs/s3.rb:143:in `process_files'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-1.0.0/lib/logstash/inputs/s3.rb:101:in `run'\norg/jruby/RubyProc.java:271:in `call'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/interval.rb:17:in `interval'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-1.0.0/lib/logstash/inputs/s3.rb:100:in `run'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:177:in `inputworker'\n/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:171:in `start_input'", :level=>:error, :file=>"logstash/pipeline.rb", :line=>"182", :method=>"inputworker"}

The credentials are working fine for the s3 output, and the policy is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::MyBucket/*",
                "arn:aws:s3:::MyBucket"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ]
        }
    ]
}

Any ideas?


(Calle Kabo) #2

The rest of the debug output:

org/jruby/ext/socket/RubyTCPSocket.java:111:in `initialize'
org/jruby/RubyIO.java:1179:in `open'
/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:763:in `connect'
org/jruby/ext/timeout/Timeout.java:126:in `timeout'
/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:763:in `connect'
/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:756:in `do_start'
/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:751:in `start'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/http/connection_pool.rb:327:in `start_session'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/http/connection_pool.rb:127:in `session_for'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/http/net_http_handler.rb:56:in `handle'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:253:in `make_sync_request'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:289:in `retry_server_errors'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/s3/region_detection.rb:11:in `retry_server_errors'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:249:in `make_sync_request'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:509:in `client_request'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:391:in `log_client_request'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:477:in `client_request'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:373:in `return_or_raise'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/client.rb:476:in `client_request'
(eval):3:in `list_objects'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/s3/object_collection.rb:298:in `list_request'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/s3/paginated_collection.rb:29:in `_each_item'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/collection/with_limit_and_next_token.rb:54:in `_each_batch'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/collection.rb:80:in `each_batch'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/core/collection.rb:47:in `each'
/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.64.0/lib/aws/s3/object_collection.rb:282:in `each'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-1.0.0/lib/logstash/inputs/s3.rb:109:in `list_new_files'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-1.0.0/lib/logstash/inputs/s3.rb:143:in `process_files'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-1.0.0/lib/logstash/inputs/s3.rb:101:in `run'
org/jruby/RubyProc.java:271:in `call'
/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/interval.rb:17:in `interval'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-1.0.0/lib/logstash/inputs/s3.rb:100:in `run'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:177:in `inputworker'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:171:in `start_input'
{:timestamp=>"2015-10-26T15:44:17.822000+0100", :message=>"Plugin is finished", :plugin=><LogStash::Inputs::S3 access_key_id=>"XXXX", secret_access_key=>"XXXX", region=>"eu-west-1", bucket=>"MyBucket", delete=>true, debug=>true, codec=><LogStash::Codecs::Plain charset=>"UTF-8">, use_ssl=>true, interval=>60, temporary_directory=>"/tmp/logstash">, :level=>:info, :file=>"logstash/plugin.rb", :line=>"61", :method=>"finished"}

(Mark Walkom) #3

What version of LS are you using?


(Calle Kabo) #4

Logstash 1.5.4


(Calle Kabo) #5

It seems someone else has gotten this error before. But using s3fog instead of s3 seems like a roundabout way of doing it. There are no http-proxys involved here anywhere.


(Calle Kabo) #6

Turns out it was a connectivity issue. Sorry for the confusion :slight_smile:


(system) #7