# Kinesis plugin not sending data to s3

**URL:** <https://discuss.elastic.co/t/kinesis-plugin-not-sending-data-to-s3/108370>\
**Category:** Logstash\
**Created:** [November 20, 2017, 11:19am UTC](https://discuss.elastic.co/t/kinesis-plugin-not-sending-data-to-s3/108370 "2017-11-20T11:19:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![hemasalma](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@hemasalma](https://discuss.elastic.co/u/hemasalma)\
**Post date:** [November 20, 2017, 11:19am UTC](https://discuss.elastic.co/t/kinesis-plugin-not-sending-data-to-s3/108370/1 "2017-11-20T11:19:56Z")

</div>

Hello,

We have created kinesis Stream and sent data to it...

logstash version is 5.4  
we are trying to send data in Stream to s3 using logstash input plugin (kinesis) and output plugin as (s3)

Below is the configuration file:

input {  
kinesis {  
kinesis\_stream\_name =\> "XXXX"  
region =\> "us-east-1"  
checkpoint\_interval\_seconds =\> 60  
application\_name =\> "logstash"  
codec =\> json { }  
}  
}  
output {  
s3{  
access\_key\_id =\> "XXXX"  
secret\_access\_key =\> "XXXXX"  
region =\> "us-east-1"  
bucket =\> "XXXX"  
codec =\> "plain"  
}  
#stdout { codec =\> rubydebug }  
}

we are getting the below error:

Nov 20, 2017 11:17:11 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker initialize  
SEVERE: Caught exception when initializing LeaseCoordinator  
com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain  
at com.amazonaws.services.kinesis.leases.impl.LeaseManager.createLeaseTableIfNotExists(LeaseManager.java:134)  
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibLeaseCoordinator.initialize(KinesisClientLibLeaseCoordinator.java:227)  
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:396)  
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:337)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:498)  
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:436)  
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:300)  
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:35)  
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)  
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)  
at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)  
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)  
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)  
at org.jruby.evaluator.ASTInterpreter.INTERPRET\_METHOD(ASTInterpreter.java:74)  
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)  
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)  
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)  
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)  
at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)  
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)  
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)  
at org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)  
at org.jruby.ast.RescueNode.interpret(RescueNode.java:116)  
at org.jruby.ast.EnsureNode.interpret(EnsureNode.java:96)  
at org.jruby.ast.BeginNode.interpret(BeginNode.java:83)  
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)  
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)  
at org.jruby.evaluator.ASTInterpreter.INTERPRET\_METHOD(ASTInterpreter.java:74)  
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)  
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)  
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)  
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)  
at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)  
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)  
at org.jruby.evaluator.ASTInterpreter.INTERPRET\_BLOCK(ASTInterpreter.java:112)  
at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)  
at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)  
at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)  
at org.jruby.runtime.Block.call(Block.java:101)  
at org.jruby.RubyProc.call(RubyProc.java:300)  
at org.jruby.RubyProc.call(RubyProc.java:230)  
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)  
at java.lang.Thread.run(Thread.java:748)  
Caused by: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain  
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131)  
at com.amazonaws.http.AmazonHttpClient.getCredentialsFromContext(AmazonHttpClient.java:781)  
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:807)  
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:702)  
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:454)  
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:416)  
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:365)  
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2048)  
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2018)  
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:878)

this error pops up even when the output we give is stdout

can you please suggest how to send data of kinesis stream to s3 using logstash.......

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 21, 2017, 4:01am UTC](https://discuss.elastic.co/t/kinesis-plugin-not-sending-data-to-s3/108370/2 "2017-11-21T04:01:23Z")

</div>

> [@hemasalma](#):
>
> Unable to load AWS credentials from any provider

Maybe you are missing [Kinesis input plugin | Logstash Reference [6.0] | Elastic](https://www.elastic.co/guide/en/logstash/6.0/plugins-inputs-kinesis.html#plugins-inputs-kinesis-profile)?

---

<div class="post-metadata">

**Author:** ![hemasalma](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@hemasalma](https://discuss.elastic.co/u/hemasalma)\
**Post date:** [November 21, 2017, 5:31am UTC](https://discuss.elastic.co/t/kinesis-plugin-not-sending-data-to-s3/108370/3 "2017-11-21T05:31:33Z")

</div>

Hi,

now i have executed the logstash by changing the config file as below

input {  
kinesis {  
kinesis\_stream\_name =\> "XXXX"  
region =\> "us-east-1"  
checkpoint\_interval\_seconds =\> 60  
application\_name =\> "logstash"  
profile =\> "~/.aws/credentials"  
type =\> "kinesis"  
codec =\> json { }  
}  
}  
output {  
stdout { codec =\> rubydebug }  
}

now the error is

Nov 21, 2017 5:17:31 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker run  
SEVERE: Unable to initialize after 20 attempts. Shutting down.  
java.lang.RuntimeException: java.lang.NullPointerException  
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:433)  
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:337)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:498)  
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:436)  
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:300)  
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:35)  
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)  
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)  
at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)  
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)  
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)  
at org.jruby.evaluator.ASTInterpreter.INTERPRET\_METHOD(ASTInterpreter.java:74)  
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)  
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)  
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)  
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)  
at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)  
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)  
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)  
at org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)  
at org.jruby.ast.RescueNode.interpret(RescueNode.java:116)  
at org.jruby.ast.EnsureNode.interpret(EnsureNode.java:96)  
at org.jruby.ast.BeginNode.interpret(BeginNode.java:83)  
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)  
at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)  
at org.jruby.evaluator.ASTInterpreter.INTERPRET\_METHOD(ASTInterpreter.java:74)  
at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)  
at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)  
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)  
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)  
at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)  
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)  
at org.jruby.evaluator.ASTInterpreter.INTERPRET\_BLOCK(ASTInterpreter.java:112)  
at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)  
at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)  
at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)  
at org.jruby.runtime.Block.call(Block.java:101)  
at org.jruby.RubyProc.call(RubyProc.java:300)  
at org.jruby.RubyProc.call(RubyProc.java:230)  
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)  
at java.lang.Thread.run(Thread.java:748)  
Caused by: java.lang.NullPointerException  
at com.amazonaws.auth.profile.ProfilesConfigFile.(ProfilesConfigFile.java:143)  
at com.amazonaws.auth.profile.ProfilesConfigFile.(ProfilesConfigFile.java:132)  
at com.amazonaws.auth.profile.ProfilesConfigFile.(ProfilesConfigFile.java:99)  
at com.amazonaws.auth.profile.ProfileCredentialsProvider.getCredentials(ProfileCredentialsProvider.java:135)  
at com.amazonaws.http.AmazonHttpClient.getCredentialsFromContext(AmazonHttpClient.java:781)  
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:807)  
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:702)  
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:454)  
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:416)  
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:365)  
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2048)  
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2018)  
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.describeTable(AmazonDynamoDBClient.java:1296)  
at com.amazonaws.services.kinesis.leases.impl.LeaseManager.tableStatus(LeaseManager.java:154)  
at com.amazonaws.services.kinesis.leases.impl.LeaseManager.createLeaseTableIfNotExists(LeaseManager.java:107)  
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibLeaseCoordinator.initialize(KinesisClientLibLeaseCoordinator.java:227)  
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:396)  
... 43 more

[2017-11-21T05:17:31,631][WARN][logstash.agent] stopping pipeline {:id=\>"main"}

do i need to modify any other things...?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 21, 2017, 5:44am UTC](https://discuss.elastic.co/t/kinesis-plugin-not-sending-data-to-s3/108370/4 "2017-11-21T05:44:50Z")

</div>

Are your credentials in the file valid?

---

<div class="post-metadata">

**Author:** ![hemasalma](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@hemasalma](https://discuss.elastic.co/u/hemasalma)\
**Post date:** [November 21, 2017, 6:11am UTC](https://discuss.elastic.co/t/kinesis-plugin-not-sending-data-to-s3/108370/5 "2017-11-21T06:11:42Z")

</div>

Yes,

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 19, 2017, 6:11am UTC](https://discuss.elastic.co/t/kinesis-plugin-not-sending-data-to-s3/108370/6 "2017-12-19T06:11:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.