# Logstash unable to start and collect logs **URL:** \ **Category:** Logstash\ **Created:** [September 13, 2022, 12:41pm UTC](https://discuss.elastic.co/t/logstash-unable-to-start-and-collect-logs/314296 "2022-09-13T12:41:46Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![khanchand](https://avatars.discourse-cdn.com/v4/letter/k/9dc877/32.png) [@khanchand](https://discuss.elastic.co/u/khanchand)\ **Post date:** [September 13, 2022, 12:41pm UTC](https://discuss.elastic.co/t/logstash-unable-to-start-and-collect-logs/314296/1 "2022-09-13T12:41:46Z")
Hi I have configure Elasticsearch, kibana & logstash on same machine. Want to receive firewall logs in logstash but facing below error. For now want to show these event on console only once received will forward in elastic. ```auto /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/firewall.conf Using bundled JDK: /usr/share/logstash/jdk WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO] 2022-09-13 08:34:39.095 [main] runner - Starting Logstash {"logstash.version"=>"8.4.1", "jruby.version"=>"jruby 9.3.6.0 (2.6.8) 2022-06-27 7a2cbcd376 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]"} [INFO] 2022-09-13 08:34:39.103 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED] Your settings are invalid. Reason: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable. [FATAL] 2022-09-13 08:34:39.142 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?] at usr.share.logstash.lib.bootstrap.environment.
(/usr/share/logstash/lib/bootstrap/environment.rb:91) ~[?:?] **firewall Configuration file is provided below:** # Input plugin that will receive the logs on port 5144 # on the specified host interface IP ###################################### input { udp { host => "172.16.41.151" port => 5140 } } # Filter plugins that will modify the logs ########################################## filter { grok { match => {"message" => "%{SYSLOG5424PRI}%{GREEDYDATA:message}" } overwrite => ["message"] } mutate { remove_field => ["@timestamp","host","@version","event","log"] } kv { field_split => " " } mutate { remove_field => ["message"] add_field => { "logdate" => "%{date} %{time}" } } date { match => ["logdate", "yyyy-MM-dd HH:mm:ss"] timezone => "America/Edmonton" target => "@timestamp" } mutate { remove_field => ["logdate","date","time"] convert => { "rcvdbyte" => "integer" } convert => { "sentbyte" => "integer" } } } # Output plugin to send the logs to Elasticsearch ################################################# output { stdout {} # Use stdout to see the output on the console during testing elasticsearch { ``` Thanks in advance ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [September 13, 2022, 12:58pm UTC](https://discuss.elastic.co/t/logstash-unable-to-start-and-collect-logs/314296/2 "2022-09-13T12:58:56Z")
> Your settings are invalid. Reason: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable. Under which account you have run LS? Does that user have permission on "/usr/share/logstash/data/? > WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. Use: /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/firewall.conf --path.settings /etc/logstash ---
**Author:** ![khanchand](https://avatars.discourse-cdn.com/v4/letter/k/9dc877/32.png) [@khanchand](https://discuss.elastic.co/u/khanchand)\ **Post date:** [September 13, 2022, 1:14pm UTC](https://discuss.elastic.co/t/logstash-unable-to-start-and-collect-logs/314296/3 "2022-09-13T13:14:55Z")
after running with specified path.settings receiving this error ```auto Using bundled JDK: /usr/share/logstash/jdk Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties 2022-09-13 09:12:08,544 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) at java.base/java.io.FileOutputStream.open0(Native Method) at java.base/java.io.FileOutputStream.open(FileOutputStream.java:293) at java.base/java.io.FileOutputStream.(FileOutputStream.java:235) at java.base/java.io.FileOutputStream.(FileOutputStream.java:155) at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:746) at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:716) at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:114) at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:100) at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:217) at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:146) at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:62) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122) at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1120) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1045) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1037) at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:651) at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:247) at org.logstash.log.LogstashConfigurationFactory.getConfiguration(LogstashConfigurationFactory.java:68) at org.logstash.log.LogstashConfigurationFactory.getConfiguration(LogstashConfigurationFactory.java:40) at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:302) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:465) at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:323) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:695) at org.apache.logging.log4j.core.LoggerContext.setConfigLocation(LoggerContext.java:684) at org.logstash.log.LoggerExt.reconfigure(LoggerExt.java:184) at org.logstash.log.LoggerExt$INVOKER$s$1$0$reconfigure.call(LoggerExt$INVOKER$s$1$0$reconfigure.gen) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:372) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:175) at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316) at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72) at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116) at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136) at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66) at org.jruby.runtime.Block.call(Block.java:147) at org.jruby.RubyProc.call(RubyProc.java:272) at org.jruby.RubyProc$INVOKER$i$call.call(RubyProc$INVOKER$i$call.gen) at org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:208) at usr.share.logstash.logstash_minus_core.lib.logstash.settings.RUBY$block$post_process$1(/usr/share/logstash/logstash-core/lib/logstash/settings.rb:198) ``` ---
**Author:** ![khanchand](https://avatars.discourse-cdn.com/v4/letter/k/9dc877/32.png) [@khanchand](https://discuss.elastic.co/u/khanchand)\ **Post date:** [September 13, 2022, 1:15pm UTC](https://discuss.elastic.co/t/logstash-unable-to-start-and-collect-logs/314296/4 "2022-09-13T13:15:39Z")
```auto 2022-09-13 09:12:08,608 main ERROR Null object returned for RollingFile in Appenders. 2022-09-13 09:12:08,608 main ERROR Null object returned for RollingFile in Appenders. 2022-09-13 09:12:08,609 main ERROR Unable to locate appender "plain_rolling" for logger config "root" 2022-09-13 09:12:08,609 main ERROR Unable to locate appender "plain_rolling_slowlog" for logger config "slowlog" 2022-09-13 09:12:08,610 main ERROR Unable to locate appender "deprecation_plain_rolling" for logger config "org.logstash.deprecation, deprecation" 2022-09-13 09:12:08,610 main ERROR Unable to locate appender "deprecation_plain_rolling" for logger config "deprecation" [2022-09-13T09:12:08,631][INFO][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties [2022-09-13T09:12:08,642][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.4.1", "jruby.version"=>"jruby 9.3.6.0 (2.6.8) 2022-06-27 7a2cbcd376 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]"} [2022-09-13T09:12:08,644][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED] Your settings are invalid. Reason: Path "/var/lib/logstash" must be a writable directory. It is not writable. [2022-09-13T09:12:08,664][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?] at usr.share.logstash.lib.bootstrap.environment.
(/usr/share/logstash/lib/bootstrap/environment.rb:91) ~[?:?] [qds@localhost ~]$ ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 13, 2022, 1:24pm UTC](https://discuss.elastic.co/t/logstash-unable-to-start-and-collect-logs/314296/5 "2022-09-13T13:24:36Z")
How you installed Logstash and how you are running? As a service? Did you installed it using any package manager? Did you run it before as the root user or using sudo? The errors you shared are permission errors, you will need to fix the permissions for the user you are running Logstash. ---
**Author:** ![khanchand](https://avatars.discourse-cdn.com/v4/letter/k/9dc877/32.png) [@khanchand](https://discuss.elastic.co/u/khanchand)\ **Post date:** [September 13, 2022, 1:29pm UTC](https://discuss.elastic.co/t/logstash-unable-to-start-and-collect-logs/314296/6 "2022-09-13T13:29:09Z")
Yes, I have run this as a root/sudo user, and have installed logstash as a service. ```auto [root@localhost ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/firewall.conf Using bundled JDK: /usr/share/logstash/jdk WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN] 2022-09-13 09:27:14.266 [main] runner - NOTICE: Running Logstash as superuser is not recommended and won't b e allowed in the future. Set 'allow_superuser' to 'false' to avoid startup errors in future releases. [INFO] 2022-09-13 09:27:14.281 [main] runner - Starting Logstash {"logstash.version"=>"8.4.1", "jruby.version"=>"jr uby 9.3.6.0 (2.6.8) 2022-06-27 7a2cbcd376 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]"} [INFO] 2022-09-13 09:27:14.284 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfi le.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Dj ava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.comp iler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add- exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-U NNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java .base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.m anagement=ALL-UNNAMED] [WARN] 2022-09-13 09:27:14.589 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [FATAL] 2022-09-13 09:27:14.596 [LogStash::Runner] runner - Logstash could not be started because there is already a nother instance using the configured data directory. If you wish to run multiple instances, you must change the "pa th.data" setting. [FATAL] 2022-09-13 09:27:14.599 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (Syst emExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?] at usr.share.logstash.lib.bootstrap.environment.
(/usr/share/logstash/lib/bootstrap/environment.rb: ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 13, 2022, 1:36pm UTC](https://discuss.elastic.co/t/logstash-unable-to-start-and-collect-logs/314296/7 "2022-09-13T13:36:30Z")
There is another instance running: > [FATAL] 2022-09-13 09:27:14.596 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting. Are you trying to run Logstash as a service, with `systemctl`? Your main issue is that you run Logstash as the root user which is not recommend because it will break the permissions if you try to run it later as a service. If you want to run logstash as a service using systemctl you will need to fix the permissions for the logstash directories. For this you will need to stop all the instances you are running and start it as a service, look at the logs and see what file or path it does not have permission and then fix the permission. ---
**Author:** ![khanchand](https://avatars.discourse-cdn.com/v4/letter/k/9dc877/32.png) [@khanchand](https://discuss.elastic.co/u/khanchand)\ **Post date:** [September 13, 2022, 1:41pm UTC](https://discuss.elastic.co/t/logstash-unable-to-start-and-collect-logs/314296/8 "2022-09-13T13:41:44Z")
yes I am running as a systemctl service. ```auto logstash.service - logstash Loaded: loaded (/usr/lib/systemd/system/logstash.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2022-09-13 08:23:39 EDT; 1h 17min ago Main PID: 73595 (java) Tasks: 64 (limit: 204125) Memory: 1.0G CGroup: /system.slice/logstash.service └─73595 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.> Sep 13 08:24:05 localhost.localdomain logstash[73595]: [2022-09-13T08:24:05,266][WARN][logstash.outputs.elasticsearch][main] Elasticsearch Output configure> Sep 13 08:24:05 localhost.localdomain logstash[73595]: [2022-09-13T08:24:05,285][WARN][logstash.filters.grok][main] ECS v8 support is a preview of the > Sep 13 08:24:05 localhost.localdomain logstash[73595]: [2022-09-13T08:24:05,319][INFO][logstash.outputs.elasticsearch][main] Using a default mapping templa> Sep 13 08:24:05 localhost.localdomain logstash[73595]: [2022-09-13T08:24:05,460][INFO][logstash.filters.kv][main] ECS compatibility is enabled but `t> Sep 13 08:24:05 localhost.localdomain logstash[73595]: [2022-09-13T08:24:05,527][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"m> Sep 13 08:24:06 localhost.localdomain logstash[73595]: [2022-09-13T08:24:06,614][INFO][logstash.javapipeline][main] Pipeline Java execution initializat> Sep 13 08:24:06 localhost.localdomain logstash[73595]: [2022-09-13T08:24:06,638][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"m> Sep 13 08:24:06 localhost.localdomain logstash[73595]: [2022-09-13T08:24:06,691][INFO][logstash.agent] Pipelines running {:count=>1, :running_pi> Sep 13 08:24:06 localhost.localdomain logstash[73595]: [2022-09-13T08:24:06,706][INFO][logstash.inputs.udp][main][c442eb5b62adae489b2531c4f3e742607e0> Sep 13 08:24:06 localhost.localdomain logstash[73595]: [2022-09-13T08:24:06,737][INFO][logstash.inputs.udp][main][c442eb5b62adae489b2531c4f3e742607e0> ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 13, 2022, 1:46pm UTC](https://discuss.elastic.co/t/logstash-unable-to-start-and-collect-logs/314296/9 "2022-09-13T13:46:15Z")
And Is the service running? It is confusing now, if you are already running Logstash as a service you **should not** try to run it again in the same server using the command line. If the service is not running, please share the logs from the service, all the logs you shared before were from Logstash running from the command line. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [September 13, 2022, 2:08pm UTC](https://discuss.elastic.co/t/logstash-unable-to-start-and-collect-logs/314296/10 "2022-09-13T14:08:44Z")
@leandrojmp I think he has active the service or LS process is still active, I would say the service is active. > Active: active (running) @khanchand Only one Logstah instance can be active. Check with ps -ef command ---
**Author:** ![khanchand](https://avatars.discourse-cdn.com/v4/letter/k/9dc877/32.png) [@khanchand](https://discuss.elastic.co/u/khanchand)\ **Post date:** [September 14, 2022, 5:24am UTC](https://discuss.elastic.co/t/logstash-unable-to-start-and-collect-logs/314296/11 "2022-09-14T05:24:15Z")
Let me do this and will let you know, if issue resolved. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 12, 2022, 5:25am UTC](https://discuss.elastic.co/t/logstash-unable-to-start-and-collect-logs/314296/12 "2022-10-12T05:25:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.