Hello Community
I have an issue in Logstash kinesis input plugin,
when I run
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kinesis.conf
I can see logs are coming from AWS and ingested into my Elasticsearch,
but when I run logstash as service
systemctl restart logstash
I can't see any logs into my Elasticsearch.
when I checked into my logs I can only see this error
[2023-11-29T11:20:23,297][ERROR][com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker][main][5053df8311ac6befc2d7b3d436772faa0ef8e49a18f0fcefa5caccd8ea2a57bf] Unable to initialize after 20 attempts. Shutting down.
java.lang.RuntimeException: java.lang.IllegalArgumentException: profile file cannot be null
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:773) ~[amazon-kinesis-client-1.15.0.jar:?]
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:672) [amazon-kinesis-client-1.15.0.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:329) [jruby.jar:?]
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:193) [jruby.jar:?]
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32) [jruby.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:474) [jruby.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195) [jruby.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:350) [jruby.jar:?]
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66) [jruby.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:82) [jruby.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201) [jruby.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188) [jruby.jar:?]
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:220) [jruby.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:495) [jruby.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244) [jruby.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:318) [jruby.jar:?]
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66) [jruby.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:82) [jruby.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201) [jruby.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188) [jruby.jar:?]
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:220) [jruby.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:495) [jruby.jar:?]
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244) [jruby.jar:?]
at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:318) [jruby.jar:?]
at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66) [jruby.jar:?]
at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116) [jruby.jar:?]
at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136) [jruby.jar:?]
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66) [jruby.jar:?]
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58) [jruby.jar:?]
at org.jruby.runtime.Block.call(Block.java:143) [jruby.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:352) [jruby.jar:?]
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110) [jruby.jar:?]
at java.lang.Thread.run(Thread.java:840) [?:?]
Bytheway I see this error when I run both commands.
also here is my config file of the kinesis input
input {
kinesis {
kinesis_stream_name => "aws-logs"
region => "ap-northeast-1"
profile => "default"
codec => cloudwatch_logs
}
}
filter {
json {
source => "message"
}
}
output {
# stdout { codec => rubydebug }
elasticsearch {
hosts => ["https://******:9200", "https://******:9200", "https://******:9200" ]
index => "awswaf-%{+YYYY.MM.dd}"
user => "test"
password => "******"
ssl_enabled => true
ssl_certificate_authorities => "/etc/logstash/elasticsearch-ca.pem"
}
}
I am also using credential file ~.aws/credentials which is located on my root user and it looks like this
[default]
aws_access_key_id = **********************
aws_secret_access_key = *******************
is there is anything I'm using any who can help me please.
Thanks.