Could not find logstash.yml

atulrana20 · September 25, 2023, 2:13pm

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

carly.richmond · September 25, 2023, 4:37pm

Hi @atulrana20,

Welcome to the community! Can you give a bit more information on what you are doing that is triggering this error?

It looks like you are trying to run Logstash, and it is unable to find the configuration file. However you have tagged it with the Elasticsearch rather than Logstash topic.

If you're trying to run Logstash can you either:

Try specifying the --path.settings option in your command with the path to your logstash.yml file?
Run your logstash command within the same folder that contains you logstash.yml file.

atulrana20 · September 25, 2023, 5:16pm

I tried with the command with path setting and still getting error

./logstash -f /etc/logstash/conf.d/syslog.conf -- path.setting

atulrana20 · September 25, 2023, 5:39pm

I used below command and got seperate error

sudo ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf

ERROR: Failed to parse YAML file "/etc/logstash/logstash.yml". Please confirm if the YAML structure is valid (e.g. look for incorrect usage of whitespace or indentation). Aborting... parser_error=>(): expected , but found '' while parsing a block mapping at line 190 column 2

carly.richmond · September 26, 2023, 11:24am

Thanks for sharing @atulrana20. It looks like you have invalid YAML in your configuration, potentially on line 190 column 2. Can you double check your use of indentation and whitespace in the file?

Feel free to share your YAML configuration and we can take a look.

atulrana20 · September 26, 2023, 12:55pm

Hello @carly.richmond I am attaching the code around line 190

The simple way to see this is to prepend each `name` with a `-`, and keep

all associated variables under the `name` they are associated with, and

above the next, like this:

modules:

name: MODULE_NAME
var.PLUGINTYPE1.PLUGINNAME1.KEY1: VALUE
var.PLUGINTYPE1.PLUGINNAME1.KEY2: VALUE
var.PLUGINTYPE2.PLUGINNAME1.KEY1: VALUE
var.PLUGINTYPE3.PLUGINNAME3.KEY1: VALUE

Module variable names must be in the format of

var.PLUGIN_TYPE.PLUGIN_NAME.KEY

carly.richmond · September 26, 2023, 1:00pm

Hi @atulrana20,

Can you share the config as a code snippet? I'm not sure if the forum has added some formatting as you shouldn't have bullet points in your config, and I would expect the first 2 lines to be commented out with a # comment statement.

atulrana20 · September 26, 2023, 1:06pm

carly.richmond · September 26, 2023, 1:12pm

@atulrana20 thanks for sharing. For future reference a code snippet is easier to read than an image.

It looks like you don't have any specific modules defined. Can you comment out the full modules section you have above and see if that fixes the problem?

atulrana20 · September 27, 2023, 8:31am

Hello @carly.richmond, Thank you for help but now we get another error. can you please look at it.

root@Logstashtest001:/usr/share/logstash/bin# sudo ./logstash --path.settings /etc/logstash/  -f /etc/logstash/conf.d/syslog.conf
Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2023-09-27T08:24:24,049][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-09-27T08:24:24,160][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.10.2", "jruby.version"=>"jruby 9.4.2.0 (3.1.0) 2023-03-08 90d2913fda OpenJDK 64-Bit Server VM 17.0.8+7 on 17.0.8+7 +indy +jit [x86_64-linux]"}
[2023-09-27T08:24:24,248][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-09-27T08:24:27,343][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-09-27T08:24:36,582][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-09-27T08:24:42,231][INFO ][org.reflections.Reflections] Reflections took 1911 ms to scan 1 urls, producing 132 keys and 464 values
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/microsoft-logstash-output-azure-loganalytics-1.0.0/lib/logstash/outputs/microsoft-logstash-output-azure-loganalytics.rb:5: warning: parentheses after method name is interpreted as an argument list, not a decomposed argument
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/microsoft-logstash-output-azure-loganalytics-1.0.0/lib/logstash/logAnalyticsClient/logStashAutoResizeBuffer.rb:3: warning: parentheses after method name is interpreted as an argument list, not a decomposed argument
[2023-09-27T08:24:45,973][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (SyntaxError) /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/mime-types-2.6.2/lib/mime/types/logger.rb:26: _1 is reserved for numbered parameter\n      def initialize(_1, _2 = nil, _3 = nil)\n", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1318)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:452)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:452)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:561)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:516)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110)", "java.base/java.lang.Thread.run(Thread.java:833)"]}
[2023-09-27T08:24:46,138][INFO ][logstash.runner          ] Logstash shut down.
[2023-09-27T08:24:46,210][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:795) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:758) ~[jruby.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]

carly.richmond · September 27, 2023, 9:33am

That potentially looks like an issue with either a plugin configuration that you've added or a plugin has been used but not installed.

Can you please share your updated configuration as code?

atulrana20 · September 27, 2023, 11:22am

Hello @carly.richmond, I have installed microsoft-logstash-output-azure-loganalytics plugin to connect to sentinel. It was sucessful as shown below. I am also attaching the config file.

root@Logstashtest001:/usr/share/logstash/bin#  sudo ./logstash-plugin install microsoft-logstash-output-azure-loganalytics
Using bundled JDK: /usr/share/logstash/jdk
Validating microsoft-logstash-output-azure-loganalytics
Unable to download data from https://rubygems.org - Connection reset (https://rubygems.org/latest_specs.4.8.gz)
ERROR: Installation aborted, verification failed for microsoft-logstash-output-azure-loganalytics
root@Logstashtest001:/usr/share/logstash/bin#  sudo ./logstash-plugin install microsoft-logstash-output-azure-loganalytics
Using bundled JDK: /usr/share/logstash/jdk
Validating microsoft-logstash-output-azure-loganalytics
Resolving mixin dependencies
Installing microsoft-logstash-output-azure-loganalytics
Installation successful```

atulrana20 · September 27, 2023, 11:23am

Also pasting the config file

input {
    tcp {
        port => "514"
        type => syslog
    }
}
 filter {
}
output {
    microsoft-logstash-output-azure-loganalytics {
      workspace_id => "********************"
      workspace_key => "*****************=="
      custom_log_table_name => "logstash_514"
    }
}

carly.richmond · September 27, 2023, 11:29am

ERROR: Installation aborted, verification failed for microsoft-logstash-output-azure-loganalytics

Did your install attempt initially fail? Can you try uninstalling and reinstalling the plugin?

atulrana20 · September 27, 2023, 12:36pm

i did unstall and install again the plugin, which worked fine.

root@Logstashtest001:/usr/share/logstash/bin# sudo ./logstash-plugin remove microsoft-logstash-output-azure-loganalytics
Using bundled JDK: /usr/share/logstash/jdk
Resolving dependencies......................................................................................................................................................................................................................................................................................................................................................................................................................................
Resolving dependencies.....................................................................................................................................................................................................................................................................................................................................................................................................................................
Successfully removed microsoft-logstash-output-azure-loganalytics
root@Logstashtest001:/usr/share/logstash/bin# sudo ./logstash-plugin install microsoft-logstash-output-azure-loganalytics
Using bundled JDK: /usr/share/logstash/jdk
Validating microsoft-logstash-output-azure-loganalytics
Resolving mixin dependencies
Installing microsoft-logstash-output-azure-loganalytics
Installation successful```

atulrana20 · September 27, 2023, 12:36pm

Still I get the error

root@Logstashtest001:/usr/share/logstash/bin# sudo ./logstash --path.settings /etc/logstash/  -f /etc/logstash/conf.d/syslog.conf
Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2023-09-27T12:31:52,097][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-09-27T12:31:52,117][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.10.2", "jruby.version"=>"jruby 9.4.2.0 (3.1.0) 2023-03-08 90d2913fda OpenJDK 64-Bit Server VM 17.0.8+7 on 17.0.8+7 +indy +jit [x86_64-linux]"}
[2023-09-27T12:31:52,157][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-09-27T12:31:55,148][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-09-27T12:32:05,887][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-09-27T12:32:11,215][INFO ][org.reflections.Reflections] Reflections took 1753 ms to scan 1 urls, producing 132 keys and 464 values
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/microsoft-logstash-output-azure-loganalytics-1.0.0/lib/logstash/outputs/microsoft-logstash-output-azure-loganalytics.rb:5: warning: parentheses after method name is interpreted as an argument list, not a decomposed argument
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/microsoft-logstash-output-azure-loganalytics-1.0.0/lib/logstash/logAnalyticsClient/logStashAutoResizeBuffer.rb:3: warning: parentheses after method name is interpreted as an argument list, not a decomposed argument
[2023-09-27T12:32:15,349][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (SyntaxError) /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/mime-types-2.6.2/lib/mime/types/logger.rb:26: _1 is reserved for numbered parameter\n      def initialize(_1, _2 = nil, _3 = nil)\n", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1318)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:452)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:452)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:561)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:516)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110)", "java.base/java.lang.Thread.run(Thread.java:833)"]}
[2023-09-27T12:32:15,581][INFO ][logstash.runner          ] Logstash shut down.
[2023-09-27T12:32:15,650][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:795) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:758) ~[jruby.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90)

atulrana20 · September 29, 2023, 7:29am

Hello @carly.richmond, The issue has been resolved by downgrading the version of logstash that works with Microsoft plugin. Thanks for the help.

system · October 27, 2023, 7:29am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.