Logstash unable to start and collect logs

khanchand · September 13, 2022, 12:41pm

Hi I have configure Elasticsearch, kibana & logstash on same machine. Want to receive firewall logs in logstash but facing below error. For now want to show these event on console only once received will forward in elastic.

 /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/firewall.conf
Using bundled JDK: /usr/share/logstash/jdk
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-09-13 08:34:39.095 [main] runner - Starting Logstash {"logstash.version"=>"8.4.1", "jruby.version"=>"jruby 9.3.6.0 (2.6.8) 2022-06-27 7a2cbcd376 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]"}
[INFO ] 2022-09-13 08:34:39.103 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
Your settings are invalid. Reason: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.
[FATAL] 2022-09-13 08:34:39.142 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:91) ~[?:?]

**firewall Configuration file is provided below:**

# Input plugin that will receive the logs on port 5144
# on the specified host interface IP
######################################

input {
 udp {
    host => "172.16.41.151"
    port => 5140
  }
}

# Filter plugins that will modify the logs
##########################################

filter {
  grok {
    match => {"message" => "%{SYSLOG5424PRI}%{GREEDYDATA:message}" }
    overwrite => [ "message" ]
  }
  mutate {
    remove_field => ["@timestamp","host","@version","event","log"]
  }
  kv {
    field_split => " "
  }
  mutate {
    remove_field => ["message"]
    add_field => { "logdate" => "%{date} %{time}" }
  }
  date {
    match => [ "logdate", "yyyy-MM-dd HH:mm:ss" ]
    timezone => "America/Edmonton"
    target => "@timestamp"
  }
  mutate {
    remove_field => ["logdate","date","time"]
    convert => { "rcvdbyte" => "integer" }
    convert => { "sentbyte" => "integer" }
  }
}

# Output plugin to send the logs to Elasticsearch
#################################################

output {
  stdout {}    # Use stdout to see the output on the console during testing
  elasticsearch {

Thanks in advance

Rios · September 13, 2022, 12:58pm

Your settings are invalid. Reason: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.

Under which account you have run LS? Does that user have permission on "/usr/share/logstash/data/?

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash.

Use:
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/firewall.conf --path.settings /etc/logstash

khanchand · September 13, 2022, 1:14pm

after running with specified path.settings receiving this error

Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
2022-09-13 09:12:08,544 main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
        at java.base/java.io.FileOutputStream.open0(Native Method)
        at java.base/java.io.FileOutputStream.open(FileOutputStream.java:293)
        at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:235)
        at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:155)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:746)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:716)
        at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:114)
        at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:100)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:217)
        at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:146)
        at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:62)
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1120)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1045)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1037)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:651)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:247)
        at org.logstash.log.LogstashConfigurationFactory.getConfiguration(LogstashConfigurationFactory.java:68)
        at org.logstash.log.LogstashConfigurationFactory.getConfiguration(LogstashConfigurationFactory.java:40)
        at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:302)
        at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:465)
        at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:323)
        at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:695)
        at org.apache.logging.log4j.core.LoggerContext.setConfigLocation(LoggerContext.java:684)
        at org.logstash.log.LoggerExt.reconfigure(LoggerExt.java:184)
        at org.logstash.log.LoggerExt$INVOKER$s$1$0$reconfigure.call(LoggerExt$INVOKER$s$1$0$reconfigure.gen)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:372)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:175)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
        at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)
        at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)
        at org.jruby.runtime.Block.call(Block.java:147)
        at org.jruby.RubyProc.call(RubyProc.java:272)
        at org.jruby.RubyProc$INVOKER$i$call.call(RubyProc$INVOKER$i$call.gen)
        at org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:208)
        at usr.share.logstash.logstash_minus_core.lib.logstash.settings.RUBY$block$post_process$1(/usr/share/logstash/logstash-core/lib/logstash/settings.rb:198)

khanchand · September 13, 2022, 1:15pm

2022-09-13 09:12:08,608 main ERROR Null object returned for RollingFile in Appenders.
2022-09-13 09:12:08,608 main ERROR Null object returned for RollingFile in Appenders.
2022-09-13 09:12:08,609 main ERROR Unable to locate appender "plain_rolling" for logger config "root"
2022-09-13 09:12:08,609 main ERROR Unable to locate appender "plain_rolling_slowlog" for logger config "slowlog"
2022-09-13 09:12:08,610 main ERROR Unable to locate appender "deprecation_plain_rolling" for logger config "org.logstash.deprecation, deprecation"
2022-09-13 09:12:08,610 main ERROR Unable to locate appender "deprecation_plain_rolling" for logger config "deprecation"
[2022-09-13T09:12:08,631][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2022-09-13T09:12:08,642][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.4.1", "jruby.version"=>"jruby 9.3.6.0 (2.6.8) 2022-06-27 7a2cbcd376 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]"}
[2022-09-13T09:12:08,644][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
Your settings are invalid. Reason: Path "/var/lib/logstash" must be a writable directory. It is not writable.
[2022-09-13T09:12:08,664][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:91) ~[?:?]
[qds@localhost ~]$

leandrojmp · September 13, 2022, 1:24pm

How you installed Logstash and how you are running? As a service?

Did you installed it using any package manager? Did you run it before as the root user or using sudo?

The errors you shared are permission errors, you will need to fix the permissions for the user you are running Logstash.

khanchand · September 13, 2022, 1:29pm

Yes, I have run this as a root/sudo user, and have installed logstash as a service.

[root@localhost ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/firewall.conf
Using bundled JDK: /usr/share/logstash/jdk
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify                                          the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which                                          logs errors to the console
[WARN ] 2022-09-13 09:27:14.266 [main] runner - NOTICE: Running Logstash as superuser is not recommended and won't b                                         e allowed in the future. Set 'allow_superuser' to 'false' to avoid startup errors in future releases.
[INFO ] 2022-09-13 09:27:14.281 [main] runner - Starting Logstash {"logstash.version"=>"8.4.1", "jruby.version"=>"jr                                         uby 9.3.6.0 (2.6.8) 2022-06-27 7a2cbcd376 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]"}
[INFO ] 2022-09-13 09:27:14.284 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfi                                         le.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Dj                                         ava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true,                                          -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.comp                                         iler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-                                         exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-U                                         NNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java                                         .base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.m                                         anagement=ALL-UNNAMED]
[WARN ] 2022-09-13 09:27:14.589 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or                                          command line options are specified
[FATAL] 2022-09-13 09:27:14.596 [LogStash::Runner] runner - Logstash could not be started because there is already a                                         nother instance using the configured data directory.  If you wish to run multiple instances, you must change the "pa                                         th.data" setting.
[FATAL] 2022-09-13 09:27:14.599 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (Syst                                         emExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
       at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
       at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
       at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:

leandrojmp · September 13, 2022, 1:36pm

There is another instance running:

[FATAL] 2022-09-13 09:27:14.596 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.

Are you trying to run Logstash as a service, with systemctl?

Your main issue is that you run Logstash as the root user which is not recommend because it will break the permissions if you try to run it later as a service.

If you want to run logstash as a service using systemctl you will need to fix the permissions for the logstash directories.

For this you will need to stop all the instances you are running and start it as a service, look at the logs and see what file or path it does not have permission and then fix the permission.

khanchand · September 13, 2022, 1:41pm

yes I am running as a systemctl service.

 logstash.service - logstash
   Loaded: loaded (/usr/lib/systemd/system/logstash.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-09-13 08:23:39 EDT; 1h 17min ago
 Main PID: 73595 (java)
    Tasks: 64 (limit: 204125)
   Memory: 1.0G
   CGroup: /system.slice/logstash.service
           └─73595 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.>

Sep 13 08:24:05 localhost.localdomain logstash[73595]: [2022-09-13T08:24:05,266][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configure>
Sep 13 08:24:05 localhost.localdomain logstash[73595]: [2022-09-13T08:24:05,285][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the >
Sep 13 08:24:05 localhost.localdomain logstash[73595]: [2022-09-13T08:24:05,319][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping templa>
Sep 13 08:24:05 localhost.localdomain logstash[73595]: [2022-09-13T08:24:05,460][INFO ][logstash.filters.kv      ][main] ECS compatibility is enabled but `t>
Sep 13 08:24:05 localhost.localdomain logstash[73595]: [2022-09-13T08:24:05,527][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"m>
Sep 13 08:24:06 localhost.localdomain logstash[73595]: [2022-09-13T08:24:06,614][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initializat>
Sep 13 08:24:06 localhost.localdomain logstash[73595]: [2022-09-13T08:24:06,638][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"m>
Sep 13 08:24:06 localhost.localdomain logstash[73595]: [2022-09-13T08:24:06,691][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pi>
Sep 13 08:24:06 localhost.localdomain logstash[73595]: [2022-09-13T08:24:06,706][INFO ][logstash.inputs.udp      ][main][c442eb5b62adae489b2531c4f3e742607e0>
Sep 13 08:24:06 localhost.localdomain logstash[73595]: [2022-09-13T08:24:06,737][INFO ][logstash.inputs.udp      ][main][c442eb5b62adae489b2531c4f3e742607e0>

leandrojmp · September 13, 2022, 1:46pm

And Is the service running?

It is confusing now, if you are already running Logstash as a service you should not try to run it again in the same server using the command line.

If the service is not running, please share the logs from the service, all the logs you shared before were from Logstash running from the command line.

Rios · September 13, 2022, 2:08pm

@leandrojmp I think he has active the service or LS process is still active, I would say the service is active.

Active: active (running)

@khanchand Only one Logstah instance can be active.
Check with ps -ef command

khanchand · September 14, 2022, 5:24am

Let me do this and will let you know, if issue resolved.

system · October 12, 2022, 5:25am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.