Logstash's Kinesis input plugin doesn't able to access aws credentials file

vnallamothu · December 6, 2018, 9:03pm

I was trying to send my logs from AWS Kinesis Firehose to my self managed Elasticsearch cluster through logstash and logstash's kinesis input plugin.

My conf file looks like the following:

input {
        kinesis {
                kinesis_stream_name => "Findings-monitor"
                region => "us-west-2"
                profile => "~/.aws/credentials"
                codec => "json"
         }
}
output {
        elasticsearch {
                hosts => "XX.XX.XX.XX:9200"
                index => "sample-index"
                             }
           }

I've added my aws credentials to the "~/.aws/credentials" file and also installed kinesis input plugin. But, when I ran the bin/logstash -f /etc/logstash/conf.d/test.conf --path.settings /etc/logstash command, I'm getting the following error:

SEVERE: Unable to initialize after 20 attempts. Shutting down.
java.lang.RuntimeException: java.lang.IllegalArgumentException: profile file cannot be null
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:664)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:564)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:438)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:302)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:339)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)
        at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)
        at org.jruby.runtime.Block.call(Block.java:124)
        at org.jruby.RubyProc.call(RubyProc.java:289)
        at org.jruby.RubyProc.call(RubyProc.java:246)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: profile file cannot be null

Can anyone explain whats my fault?

Raywon_Teja_Kari · December 7, 2018, 9:16pm

Hi,

here you need to use the profile name defined in your credentials file, not the path to this, if I'm not wrong...

for example: default is the profile name here

[default]
aws_access_key_id = XX
aws_secret_access_key = XX

vnallamothu · December 7, 2018, 9:48pm

Hi,

Thank you for the reply. Even if I changed the profile value to the default, it is throwing the same error. Also, I tried exporting those values as environmental variables and deleted the profile => default line. Still no luck.

Raywon_Teja_Kari · December 7, 2018, 9:56pm

What error did you get when you used profile => "default"?
How did you set environment variables and what did you set?
What error did you get when you used environment variables?

vnallamothu · December 7, 2018, 10:02pm

In all the cases, I got the same error that I've listed in my first post.

SEVERE: Unable to initialize after 20 attempts. Shutting down.
java.lang.RuntimeException: java.lang.IllegalArgumentException: profile file cannot be null
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:664)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:564)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:438)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:302)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:339)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)
        at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)
        at org.jruby.runtime.Block.call(Block.java:124)
        at org.jruby.RubyProc.call(RubyProc.java:289)
        at org.jruby.RubyProc.call(RubyProc.java:246)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: profile file cannot be null

I've set my ACCESS KEY ID and Secret access key ID as environmental variables. with commands:
export AWS_ACCESS_KEY_ID=your_access_key_id
export AWS_SECRET_ACCESS_KEY=your_secret_access_key

vnallamothu · December 7, 2018, 11:47pm

Do I need to create a DynamoDB table or the logstash configuration will create one? Do you have any idea?

vnallamothu · December 10, 2018, 6:09pm

if I run sudo bin/logstash -e /etc/logstash/conf.d/test.conf --path.settings /etc/logstash, instead of the parameter "-f" it is throwing the error:
[2018-12-10T16:30:17,624][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 3, column 1 (byte 76) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:incompile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:inblock in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:incompile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:inblock in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:inblock in converge_state'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:inconverge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}`

vnallamothu · December 11, 2018, 3:50pm

My mistake is that I am running the bin/logstash command with sudo privileges but didn't assign environmental variables in root user profile.

system · January 8, 2019, 3:50pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash unable to authenticate and get data from AWS kinesis stream Logstash	1	714	August 22, 2019
Profile file cannot be null Logstash error Logstash	7	478	December 28, 2023
Kinesis plugin not sending data to s3 Logstash	5	1065	December 19, 2017
Securing the aws credentials in logstash kinesis pluging Logstash	1	314	August 6, 2019
The security token included in the request is invalid. (Service: AmazonDynamoDBv2) Elasticsearch	1	6265	May 17, 2019

Logstash's Kinesis input plugin doesn't able to access aws credentials file

Related topics