Profile file cannot be null Logstash error

laale1 · November 29, 2023, 11:42am

Hello Community

I have an issue in Logstash kinesis input plugin,
when I run

 /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kinesis.conf

I can see logs are coming from AWS and ingested into my Elasticsearch,
but when I run logstash as service

systemctl restart logstash

I can't see any logs into my Elasticsearch.

when I checked into my logs I can only see this error

 [2023-11-29T11:20:23,297][ERROR][com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker][main][5053df8311ac6befc2d7b3d436772faa0ef8e49a18f0fcefa5caccd8ea2a57bf] Unable to initialize after 20 attempts. Shutting down.
java.lang.RuntimeException: java.lang.IllegalArgumentException: profile file cannot be null
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:773) ~[amazon-kinesis-client-1.15.0.jar:?]
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:672) [amazon-kinesis-client-1.15.0.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:329) [jruby.jar:?]
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:193) [jruby.jar:?]
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32) [jruby.jar:?]
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:474) [jruby.jar:?]
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195) [jruby.jar:?]
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:350) [jruby.jar:?]
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66) [jruby.jar:?]
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:82) [jruby.jar:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201) [jruby.jar:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188) [jruby.jar:?]
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:220) [jruby.jar:?]
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:495) [jruby.jar:?]
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244) [jruby.jar:?]
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:318) [jruby.jar:?]
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66) [jruby.jar:?]
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:82) [jruby.jar:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201) [jruby.jar:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188) [jruby.jar:?]
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:220) [jruby.jar:?]
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:495) [jruby.jar:?]
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244) [jruby.jar:?]
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:318) [jruby.jar:?]
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66) [jruby.jar:?]
        at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116) [jruby.jar:?]
        at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136) [jruby.jar:?]
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66) [jruby.jar:?]
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58) [jruby.jar:?]
        at org.jruby.runtime.Block.call(Block.java:143) [jruby.jar:?]
        at org.jruby.RubyProc.call(RubyProc.java:352) [jruby.jar:?]
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110) [jruby.jar:?]
        at java.lang.Thread.run(Thread.java:840) [?:?]

Bytheway I see this error when I run both commands.

also here is my config file of the kinesis input

 input {
  kinesis {
    kinesis_stream_name => "aws-logs"
    region => "ap-northeast-1"
    profile => "default"
    codec => cloudwatch_logs
  }
}

filter {
  json {
    source => "message"
  }
}

output {
#  stdout { codec => rubydebug }
  elasticsearch {
    hosts => ["https://******:9200", "https://******:9200", "https://******:9200" ]
     index => "awswaf-%{+YYYY.MM.dd}"
     user => "test"
     password => "******"
     ssl_enabled => true
     ssl_certificate_authorities => "/etc/logstash/elasticsearch-ca.pem"
   }
}

I am also using credential file ~.aws/credentials which is located on my root user and it looks like this

[default]
aws_access_key_id = **********************
aws_secret_access_key = *******************

is there is anything I'm using any who can help me please.

Thanks.

Rios · November 29, 2023, 1:55pm

LS is using logstash:logstash to run in the service mode. Give rights to logstash user or put the credentials in /etc/logstash/

laale1 · November 29, 2023, 2:01pm

hey @Rios thanks bro for the replay when I put the credentials on to the /etc/Logstash do I have to put the full path of it in the conf file or still i'm using this profile => "default"

laale1 · November 29, 2023, 2:06pm

@Rios I put the credential file in the /etc/logstash and gave him the permission but still i see the error

Caused by: java.lang.IllegalArgumentException: profile file cannot be null
        at com.amazonaws.util.ValidationUtils.assertNotNull(ValidationUtils.java:37) ~[aws-java-sdk-core-1.12.408.jar:?]
        at com.amazonaws.auth.profile.ProfilesConfigFile.<init>(ProfilesConfigFile.java:142) ~[aws-java-sdk-core-1.12.408.jar:?]
        at com.amazonaws.auth.profile.ProfilesConfigFile.<init>(ProfilesConfigFile.java:133) ~[aws-java-sdk-core-1.12.408.jar:?]
        at com.amazonaws.auth.profile.ProfilesConfigFile.<init>(ProfilesConfigFile.java:100) ~[aws-java-sdk-core-1.12.408.jar:?]

Rios · November 30, 2023, 7:24am

Forget it /etc/logstash, my mistake.
According to the documentation should be:
~/.aws/credentials - you are missing a forward slash. If you still have problem, edit /etc/default/logstash file, set:

LS_USER="root"
LS_GROUP="root"

If this works, you have a permission issue.

laale1 · November 30, 2023, 9:31am

hey @Rios thanks bro always for the replay

I did all the things you mentioned and still I'm seeing the error,

java.lang.RuntimeException: java.lang.IllegalArgumentException: profile file cannot be null
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:773) ~[amazon-kinesis-client-1.15.0.jar:?]

The crazy thing is while seeing this error, when I run this command,

 /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kinesis.conf

I can see my logs displayed or ingest in my logstash

but when I run logstash as a service and run,

systemctl restart logstash

I can't see any logs, what is wrong when I run it as a service.

laale1 · November 30, 2023, 10:50am

I finally solved the issue by also editing /lib/systemd/system/logstash.service

and changing logstash user to root

[Service]
Type=simple
User=root
Group=root

Thanks bro @Rios for the help.

system · December 28, 2023, 10:50am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.