Unable to start logstash configuration file

Charles_Yuliansen · February 10, 2020, 3:39am

this is my first time running logstash on container. Im running logstash on the same container elasticsearch + kibana. Its running on ubuntu.
i run my conf file by using

/usr/share/logstash/bin/logstash -f conf.d/logstash.conf

here is my logstash.conf :

input{
        beats{
                port=>5044
        }
}
filter
{
        grok {
                match =>{
                  "message" => "%{TIMESTAMP_ISO8601:logtimestamp}\s%{DATA:S_IP}\s%{WORD:s_method}\s%{DATA:cs_uri_stem}\s%{DATA:cs_uri_query}\s%{DATA:s_port}\s%{GREEDYDATA:log_message}"
                }
        }
        date{
        match =>["logtimestamp","yyyy-MM-dd HH:mm:ss"]
        target=>"@timestamp"
        }
}
output{
stdout{codec=>rubydebug}
        elasticsearch{
        hosts=>"elastic@localhost:9200"
        index=>"log_iis"
        user =>"*****"
        password=>"*****"
}
}

and it returning error as :

Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-02-10 03:32:59.625 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-02-10 03:32:59.632 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.5.2"}
[INFO ] 2020-02-10 03:33:00.995 [Converge PipelineAction::Create<main>] Reflections - Reflections took 24 ms to scan 1 urls, producing 20 keys and 40 values
[ERROR] 2020-02-10 03:33:01.375 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: Illegal character in scheme name at index 7: elastic@localhost:9200", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:60)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1156)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1143)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:27)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.Block.call(Block.java:129)", "org.jruby.RubyProc.call(RubyProc.java:295)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.RubyProc.call(RubyProc.java:270)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:830)"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
          create at org/logstash/execution/ConvergeResultExt.java:109
             add at org/logstash/execution/ConvergeResultExt.java:37
  converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:339
[ERROR] 2020-02-10 03:33:01.379 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
[FATAL] 2020-02-10 03:33:01.403 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
[ERROR] 2020-02-10 03:33:01.432 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit)

and respond or explanation will be appreciated so much. thank you

ropc · February 10, 2020, 6:24am

Hi @Charles_Yuliansen - The error complains about the URI value you specified in the hosts parameter (index 7 in the string corresponds to the @ character). Should not the hosts parameter value be hosts=> ["localhost:9200"] ?

Charles_Yuliansen · February 10, 2020, 6:28am

hi there thank you so much for your respond, i've tried just using localhost and 127.0.0.1 and it returning

[WARN ] 2020-02-10 04:27:29.004 [Ruby-0-Thread-4: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}

dont mind http://elastic:xxxxxx@elasticsearch:9200/ elasticsearch:9200, its just me being silly and desperate so i try everything

ropc · February 10, 2020, 6:46am

Hi @Charles_Yuliansen - You mentioned that you are running Logstash, Elasticsearch and Kibana in the same container? Can you confirm that? I would actually assume that you are running these components in different containers.

The error Elasticsearch Unreachable indicates that your Elasticsearch cluster is not reachable - it might be a problem with the resolution of the hostname you specified. The hosts parameter value should be a valid IP address or a valid hostname (and that can be resolved) pointing to your Elasticsearch cluster.

Charles_Yuliansen · February 10, 2020, 8:28am

Hi there @ropc,it took me a while to make it sure, but im pretty sure that i installed the logstash in the same container as elasticsearch.

system · March 9, 2020, 8:28am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.