Logstash's Kinesis input plugin doesn't able to access aws credentials file

I was trying to send my logs from AWS Kinesis Firehose to my self managed Elasticsearch cluster through logstash and logstash's kinesis input plugin.

My conf file looks like the following:

input {
        kinesis {
                kinesis_stream_name => "Findings-monitor"
                region => "us-west-2"
                profile => "~/.aws/credentials"
                codec => "json"
         }
}
output {
        elasticsearch {
                hosts => "XX.XX.XX.XX:9200"
                index => "sample-index"
                             }
           }

I've added my aws credentials to the "~/.aws/credentials" file and also installed kinesis input plugin. But, when I ran the bin/logstash -f /etc/logstash/conf.d/test.conf --path.settings /etc/logstash command, I'm getting the following error:

SEVERE: Unable to initialize after 20 attempts. Shutting down.
java.lang.RuntimeException: java.lang.IllegalArgumentException: profile file cannot be null
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:664)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:564)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:438)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:302)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:339)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)
        at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)
        at org.jruby.runtime.Block.call(Block.java:124)
        at org.jruby.RubyProc.call(RubyProc.java:289)
        at org.jruby.RubyProc.call(RubyProc.java:246)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: profile file cannot be null

Can anyone explain whats my fault?

Hi,

here you need to use the profile name defined in your credentials file, not the path to this, if I'm not wrong...

for example: default is the profile name here

[default]
aws_access_key_id = XX
aws_secret_access_key = XX

Hi,

Thank you for the reply. Even if I changed the profile value to the default, it is throwing the same error. Also, I tried exporting those values as environmental variables and deleted the profile => default line. Still no luck.

  • What error did you get when you used profile => "default"?
  • How did you set environment variables and what did you set?
  • What error did you get when you used environment variables?

In all the cases, I got the same error that I've listed in my first post.

SEVERE: Unable to initialize after 20 attempts. Shutting down.
java.lang.RuntimeException: java.lang.IllegalArgumentException: profile file cannot be null
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:664)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:564)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:438)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:302)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:339)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
        at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)
        at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)
        at org.jruby.runtime.Block.call(Block.java:124)
        at org.jruby.RubyProc.call(RubyProc.java:289)
        at org.jruby.RubyProc.call(RubyProc.java:246)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: profile file cannot be null

I've set my ACCESS KEY ID and Secret access key ID as environmental variables. with commands:
export AWS_ACCESS_KEY_ID=your_access_key_id
export AWS_SECRET_ACCESS_KEY=your_secret_access_key

Do I need to create a DynamoDB table or the logstash configuration will create one? Do you have any idea?

if I run sudo bin/logstash -e /etc/logstash/conf.d/test.conf --path.settings /etc/logstash, instead of the parameter "-f" it is throwing the error:
[2018-12-10T16:30:17,624][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 3, column 1 (byte 76) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:incompile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:inblock in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:incompile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:inblock in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:inblock in converge_state'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:inconverge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}`

My mistake is that I am running the bin/logstash command with sudo privileges but didn't assign environmental variables in root user profile.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.