Logstash's Kinesis input plugin doesn't able to access aws credentials file

vnallamothu · December 6, 2018, 9:03pm

I was trying to send my logs from AWS Kinesis Firehose to my self managed Elasticsearch cluster through logstash and logstash's kinesis input plugin.

My conf file looks like the following:

input {
        kinesis {
                kinesis_stream_name => "Findings-monitor"
                region => "us-west-2"
                profile => "~/.aws/credentials"
                codec => "json"
         }
}
output {
        elasticsearch {
                hosts => "XX.XX.XX.XX:9200"
                index => "sample-index"
                             }
           }

I've added my aws credentials to the "~/.aws/credentials" file and also installed kinesis input plugin. But, when I ran the bin/logstash -f /etc/logstash/conf.d/test.conf --path.settings /etc/logstash command, I'm getting the following error:

SEVERE: Unable to initialize after 20 attempts. Shutting down.
java.lang.RuntimeException: java.lang.IllegalArgumentException: profile file cannot be null
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:664)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:564)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:438)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:302)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:339)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)
        at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)
        at org.jruby.runtime.Block.call(Block.java:124)
        at org.jruby.RubyProc.call(RubyProc.java:289)
        at org.jruby.RubyProc.call(RubyProc.java:246)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: profile file cannot be null

Can anyone explain whats my fault?

Raywon_Teja_Kari · December 7, 2018, 9:16pm

Hi,

here you need to use the profile name defined in your credentials file, not the path to this, if I'm not wrong...

for example: default is the profile name here

[default]
aws_access_key_id = XX
aws_secret_access_key = XX

vnallamothu · December 7, 2018, 9:48pm

Hi,

Thank you for the reply. Even if I changed the profile value to the default, it is throwing the same error. Also, I tried exporting those values as environmental variables and deleted the profile => default line. Still no luck.

Raywon_Teja_Kari · December 7, 2018, 9:56pm

What error did you get when you used profile => "default"?
How did you set environment variables and what did you set?
What error did you get when you used environment variables?

vnallamothu · December 7, 2018, 10:02pm

In all the cases, I got the same error that I've listed in my first post.

SEVERE: Unable to initialize after 20 attempts. Shutting down.
java.lang.RuntimeException: java.lang.IllegalArgumentException: profile file cannot be null
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:664)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:564)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:438)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:302)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:339)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)
        at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)
        at org.jruby.runtime.Block.call(Block.java:124)
        at org.jruby.RubyProc.call(RubyProc.java:289)
        at org.jruby.RubyProc.call(RubyProc.java:246)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: profile file cannot be null

I've set my ACCESS KEY ID and Secret access key ID as environmental variables. with commands:
export AWS_ACCESS_KEY_ID=your_access_key_id
export AWS_SECRET_ACCESS_KEY=your_secret_access_key

vnallamothu · December 7, 2018, 11:47pm

Do I need to create a DynamoDB table or the logstash configuration will create one? Do you have any idea?

vnallamothu · December 10, 2018, 6:09pm

if I run sudo bin/logstash -e /etc/logstash/conf.d/test.conf --path.settings /etc/logstash, instead of the parameter "-f" it is throwing the error:
[2018-12-10T16:30:17,624][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 3, column 1 (byte 76) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:incompile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:inblock in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:incompile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:inblock in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:inblock in converge_state'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:inconverge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}`

vnallamothu · December 11, 2018, 3:50pm

My mistake is that I am running the bin/logstash command with sudo privileges but didn't assign environmental variables in root user profile.

system · January 8, 2019, 3:50pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.