# S3 Input Plugin randomly fails

**URL:** <https://discuss.elastic.co/t/s3-input-plugin-randomly-fails/115342>\
**Category:** Logstash\
**Created:** [January 12, 2018, 5:14pm UTC](https://discuss.elastic.co/t/s3-input-plugin-randomly-fails/115342 "2018-01-12T17:14:03Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![kpaulsenegain](https://avatars.discourse-cdn.com/v4/letter/k/d2c977/32.png) [@kpaulsenegain](https://discuss.elastic.co/u/kpaulsenegain)\
**Post date:** [January 12, 2018, 5:14pm UTC](https://discuss.elastic.co/t/s3-input-plugin-randomly-fails/115342/1 "2018-01-12T17:14:03Z")

</div>

I'm using Logstash version 6.0.1 and S3 plugin logstash-input 3.2.0. I'm trying to process Web Server IIS Access logs files from a S3 bucket. The files are large 2.4 GB. The issue is that it works basically but randomly the Logstash will stop being able to process files from the bucket and in the logs I get

[2018-01-12T09:10:55,462][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.  
Pipeline\_id:main  
Plugin: \<LogStash::Inputs::S3 access\_key\_id=\>"AKIAJFN3SD4G4GILLZPQ", bucket=\>"egain-cloud-test", region=\>"us-east-1", secret\_access\_key=\>"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", type=\>"testlog", sincedb\_path=\>"/home/elastic/sincedb/test\_sincedb.db", prefix=\>"2017/04/24/u\_ex", backup\_add\_prefix=\>"archived-", backup\_to\_bucket=\>"egain-cloud-test", interval=\>120, delete=\>true, add\_field=\>{"deploymentid"=\>"test", "pipe"=\>"iislogs"}, id=\>"26cb7e9e5a1189724f9ae62ad6a0e63027404a41c41a409125a94cf4f5a8a8d6", enable\_metric=\>true, codec=\>\<LogStash::Codecs::Plain id=\>"plain\_a405d322-89a5-4fce-9bb2-20d58a3df6e3", enable\_metric=\>true, charset=\>"UTF-8"\>, temporary\_directory=\>"/tmp/logstash"\>  
Error: Failed to open TCP connection to [egain-cloud-test.s3.amazonaws.com:443](http://egain-cloud-test.s3.amazonaws.com:443) (initialize: name or service not known)  
Exception: Seahorse::Client::NetworkingError  
Stack: org/jruby/ext/socket/RubyTCPSocket.java:137:in `initialize' org/jruby/RubyIO.java:1154:in`open'  
/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:885:in `block in connect' org/jruby/ext/timeout/Timeout.java:149:in`timeout'  
/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:883:in `connect' /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:868:in`do\_start'  
/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:863:in `start' /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/delegate.rb:83:in`method\_missing'  
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net\_http/connection\_pool.rb:281:in `start_session' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net_http/connection_pool.rb:93:in`session\_for'  
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net\_http/handler.rb:116:in `session' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net_http/handler.rb:68:in`transmit'  
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net\_http/handler.rb:42:in `call' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/content_length.rb:12:in`call'  
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3\_request\_signer.rb:88:in `call' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_request_signer.rb:23:in`call'  
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/xml/error\_handler.rb:8:in `call' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_request_signer.rb:65:in`call'  
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3\_redirects.rb:15:in `call' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:87:in`call'  
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry\_errors.rb:118:in `retry_request' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:101:in`retry\_if\_possible'  
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry\_errors.rb:89:in `call' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:118:in`retry\_request'  
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry\_errors.rb:101:in `retry_if_possible' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:89:in`call'  
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry\_errors.rb:118:in `retry_request' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:101:in`retry\_if\_possible'  
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry\_errors.rb:89:in `call' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_accelerate.rb:42:in`call'  
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3\_md5s.rb:31:in `call' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21:in`call'  
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3\_bucket\_name\_restrictions.rb:12:in `call' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_bucket_dns.rb:31:in`call'  
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/rest/handler.rb:7:in `call' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/user_agent.rb:12:in`call'

:

usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins  
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:574:in `inputworker' /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:567:in`block in start\_input'

---

<div class="post-metadata">

**Author:** ![kpaulsenegain](https://avatars.discourse-cdn.com/v4/letter/k/d2c977/32.png) [@kpaulsenegain](https://discuss.elastic.co/u/kpaulsenegain)\
**Post date:** [January 12, 2018, 5:18pm UTC](https://discuss.elastic.co/t/s3-input-plugin-randomly-fails/115342/2 "2018-01-12T17:18:02Z")

</div>

This is my logstash config  
input {  
s3 {  
access\_key\_id =\> "xxx"  
bucket =\> "test"  
region =\> "xxx"  
secret\_access\_key =\> "xxx"  
type =\> "testlog"  
sincedb\_path =\> "/home/elastic/sincedb/test\_sincedb.db"  
prefix =\> "2017/04/24/u\_ex"  
backup\_add\_prefix =\> "archived-"  
backup\_to\_bucket =\> "test"  
interval =\> 120  
delete =\> true  
add\_field =\> {"deploymentid" =\> "test"}  
add\_field =\> {"pipe" =\> "iislogs"}  
}  
}

filter {  
if [message] =~ "#" {  
drop{}  
}  
}

output {  
elasticsearch {  
hosts =\> "[http://test](http://test)"  
pipeline =\> "%{pipe}"  
index =\> "%{deploymentid}\_filebeat-%{+yyy.MM.dd}"  
}  
}

Any help would be appreciated since this seems like basic functionality that should work and not randomly just stop working. The only way I can correct it is to have to shut down logstash and then restart it

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 9, 2018, 5:18pm UTC](https://discuss.elastic.co/t/s3-input-plugin-randomly-fails/115342/3 "2018-02-09T17:18:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.