Error while starting logstash

Lipi_Lapsiwala · December 7, 2020, 6:59pm

I am trying to integrate Elastic stack (Logstash, ElasticSearch and Kibana) with IBM websphere liberty profile. I have successfully created and configured server.xml file in wlp and able to start and access the server. I was able to install and start elasticSearch and Kibana as well. However after installing and configuring logstash, I am receving below error whiletrying to start logstash:

./bin/logstash -f liberty_logstash.config
Using JAVA_HOME defined java: /home/odmadmin/IBM/ODM8104/jdk
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
Sending Logstash logs to /home/odmadmin/IBM/elastic/logstash/logstash-7.10.0/logs which is now configured via log4j2.properties
[2020-12-07T14:39:25,195][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [linux-x86_64]"}
[2020-12-07T14:39:25,698][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-12-07T14:39:26,625][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/home/odmadmin/IBM/elastic/logstash/logstash-7.10.0/liberty_logstash.config"}
[2020-12-07T14:39:26,657][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2020-12-07T14:39:27,112][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-12-07T14:39:32,033][INFO ][logstash.runner          ] Logstash shut down.
[2020-12-07T14:39:32,045][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Could someone help to figure out the solution for this error?

Note: I am using this link as a reference for the whole configuration : https://www.ibm.com/support/knowledgecenter/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/twlp_analytics_logstash.html

warkolm · December 8, 2020, 12:31am

I would start there and make sure you have configuration specified.

Lipi_Lapsiwala · December 8, 2020, 12:47am

I am doing this for the first time so I am not sure but I do have configuration specified at location "/home/odmadmin/IBM/elastic/logstash/logstash-7.10.0/" in file named "liberty_logstash.config".

below is the content of the file:

# The Lumberjack protocol has been deprecated
# and Logstash collector now operates on the new
# Beats protocol. Uncomment the sample input below
# if you need to use the Lumberjack protocol.
#
#input {
#    lumberjack {
#        port => "5043"
#        ssl_certificate => "/config/logstash.crt" 
#        ssl_key => "/config/logstash.key" 
#    }
#}

input {
    beats {
        port => "5043"
    ssl => true
        ssl_certificate => "/config/logstash.crt" 
        ssl_key => "/config/logstash.key" 
    }
}
filter {
    if [type] =~ "liberty_*" {
        if [type] == "liberty_accesslog" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
        else if [type] == "liberty_trace" {
            json {
                source => "message"
            }
        }
        else if [type] == "liberty_gc" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
        else if [type] == "liberty_message" {
            json {
                source => "message"
            }
        }
        else if [type] == "liberty_ffdc" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
        else if [type] == "liberty_audit" {
            json {
                source => "message"
                remove_field => [ "message" ]
            }
        }
    }
}
output {
    elasticsearch {
        hosts => "elasticsearch:9200"
        template => "/config/liberty_logstash_template_ibm.json"
        template_overwrite => "true"
        document_type => "_doc"
    }
}

Also I am using this command " ./bin/logstash -f liberty_logstash.config " to start logstash from this location "/home/odmadmin/IBM/elastic/logstash/logstash-7.10.0/"

warkolm · December 8, 2020, 12:48am

Try renaming that to a .conf instead? Not 100% sure that will work, but the usual standard is to use that suffix.

Lipi_Lapsiwala · December 8, 2020, 1:07am

okay it worked but I got another error:

[odmadmin@eagles logstash-7.10.0]$ ./bin/logstash -f liberty_logstash.conf
Using JAVA_HOME defined java: /home/odmadmin/IBM/ODM8104/jdk
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
Sending Logstash logs to /home/odmadmin/IBM/elastic/logstash/logstash-7.10.0/logs which is now configured via log4j2.properties
[2020-12-07T20:56:40,339][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0_261-b12 +indy +jit [linux-x86_64]"}
[2020-12-07T20:56:40,851][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-12-07T20:56:43,999][INFO ][org.reflections.Reflections] Reflections took 39 ms to scan 1 urls, producing 23 keys and 47 values 
[2020-12-07T20:56:44,348][ERROR][logstash.inputs.beats    ] Invalid setting for beats input plugin:

  input {
    beats {
      # This setting must be a path
      # File does not exist or cannot be opened /config/logstash.crt
      ssl_certificate => "/config/logstash.crt"
      ...
    }
  }
[2020-12-07T20:56:44,351][ERROR][logstash.inputs.beats    ] Invalid setting for beats input plugin:

  input {
    beats {
      # This setting must be a path
      # File does not exist or cannot be opened /config/logstash.key
      ssl_key => "/config/logstash.key"
      ...
    }
  }
[2020-12-07T20:56:44,375][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:83)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "home.odmadmin.IBM.elastic.logstash.logstash_minus_7_dot_10_dot_0.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/home/odmadmin/IBM/elastic/logstash/logstash-7.10.0/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "home.odmadmin.IBM.elastic.logstash.logstash_minus_7_dot_10_dot_0.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/home/odmadmin/IBM/elastic/logstash/logstash-7.10.0/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "home.odmadmin.IBM.elastic.logstash.logstash_minus_7_dot_10_dot_0.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/home/odmadmin/IBM/elastic/logstash/logstash-7.10.0/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "home.odmadmin.IBM.elastic.logstash.logstash_minus_7_dot_10_dot_0.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/home/odmadmin/IBM/elastic/logstash/logstash-7.10.0/logstash-core/lib/logstash/agent.rb:365)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Thread.java:748)"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
          create at org/logstash/execution/ConvergeResultExt.java:129
             add at org/logstash/execution/ConvergeResultExt.java:57
  converge_state at /home/odmadmin/IBM/elastic/logstash/logstash-7.10.0/logstash-core/lib/logstash/agent.rb:378
[2020-12-07T20:56:44,390][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle  `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
[2020-12-07T20:56:44,433][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in `create'", "org/logstash/execution/ConvergeResultExt.java:57:in `add'", "/home/odmadmin/IBM/elastic/logstash/logstash-7.10.0/logstash-core/lib/logstash/agent.rb:378:in `block in converge_state'"]}
[2020-12-07T20:56:44,463][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

I have two files "logstash.key" and "logstash.crt" at location

/home/odmadmin/IBM/elastic/logstash/logstash-7.10.0/config

I already posted the content of liberty_logstash.conf file's above. Those files were created and imported using the 2nd and the 4th steps respectively from this link

https://www.ibm.com/support/knowledgecenter/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/twlp_analytics_logstash.html

Badger · December 8, 2020, 1:08am

That's an absolute path that does not match where you say the file is (/home/odmadmin/IBM/elastic/logstash/logstash-7.10.0/).

Lipi_Lapsiwala · December 8, 2020, 1:32am

It finally started . Thank you so much for pointing out all the mistakes

system · January 5, 2021, 1:32am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.