After Installing Logstash version 8.11.4 it Exit because a System error

jcourt2006 · January 14, 2024, 6:33pm

I get the following error:

[root@app logstash]# /usr/share/logstash/bin/logstash -t --path.settings /etc/logstash
Using bundled JDK: /usr/share/logstash/jdk
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2024-01-14T13:20:37,220][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2024-01-14T13:20:37,225][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.11.4", "jruby.version"=>"jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 17.0.9+9 on 17.0.9+9 +indy +jit [x86_64-linux]"}
[2024-01-14T13:20:37,228][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true,
-Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2024-01-14T13:20:39,617][INFO ][org.reflections.Reflections] Reflections took 250 ms to scan 1 urls, producing 131 keys and 463 values
[2024-01-14T13:20:40,875][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility:
v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
Configuration OK
[2024-01-14T13:20:40,883][INFO ][logstash.runner          ] Using config.test_and_exit mode. **Config Validation Result: OK.** Exiting Logstash

[root@app logstash]# /usr/share/logstash/bin/logstash --version
Using bundled JDK: /usr/share/logstash/jdk
logstash 8.11.4

The only error I can find is in /var/log/message, namely:

Jan 14 13:28:00 app logstash[72439]: Your settings are invalid. Reason: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.

I have performed the command: chown 755 /var/lib/logstash/queue
Also I change the ownership of folder /etc/logstash from root to logstash (chown -R logstash:logstash /etc/logstash).

I enable ecs compability in the file /etc/logstash/logstash.yml

I don't have any clue why it is not working, can someone give some hints?

jcourt2006 · January 14, 2024, 7:01pm

I found the following:

[root@app logstash]# ls -ltrah /var/lib/logstash/
total 8.0K
drwxr-xr-x. 58 root     root     4.0K Jan  1 11:37 ..
drwxr-xr-x.  2      977      976    6 Jan  1 19:21 queue
drwxr-xr-x.  2      977      976    6 Jan  1 19:21 dead_letter_queue
-rw-r--r--.  1      977      976    0 Jan  1 19:21 .lock
-rw-r--r--.  1      977      976   36 Jan  1 19:21 uuid
drwxr-xr-x.  4      977      976   35 Jan  2 18:03 plugins
drwxr-xr-x.  5 logstash logstash   84 Jan  5 11:41 .

I change the ownership to Logstash and the Pipeline is working fine but when I send a log to Rsyslog I notice that Logstash restart.
the Logger command and Logstash configuration is as follow:

The logstash configuration is:

input {
  file {
    path => "/data/syslog/syslog-*.log"
    tags => ["syslog"]
#    start_position => "beginning"
  }
}

##Filter section
filter {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
  geoip {
      source => "clientip"
      ecs_compatibility => disabled
    }
}output {
  file {
    path => "/var/log/output-logs-%{+YYYYMMdd}.log"
    codec => line { format => "custom format: %{message}"}
    #codec => rubydebug
      }
  }

The output file is not created!
When checking the configuration I get the following output:

[root@app logstash]# /usr/share/logstash/bin/logstash -t --path.settings /etc/logstash
Using bundled JDK: /usr/share/logstash/jdk
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2024-01-14T13:56:43,619][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2024-01-14T13:56:43,625][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.11.4", "jruby.version"=>"jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 17.0.9+9 on 17.0.9+9 +indy +jit [x86_64-linux]"}
[2024-01-14T13:56:43,629][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true,
-Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2024-01-14T13:56:44,942][INFO ][org.reflections.Reflections] Reflections took 144 ms to scan 1 urls, producing 131 keys and 463 values
[2024-01-14T13:56:45,531][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility:
v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
Configuration OK
[2024-01-14T13:56:45,532][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

I only see the following error in the /var/log/messages:

Jan 14 13:50:31 app logstash[75355]: 2024-01-14 13:50:31,707 main ERROR Unable to locate appender "deprecation_plain_rolling" for logger config "org.logstash.deprecation, deprecation"
Jan 14 13:50:31 app logstash[75355]: 2024-01-14 13:50:31,708 main ERROR Unable to locate appender "deprecation_plain_rolling" for logger config "deprecation"

How can I solve above issue?

stephenb · January 14, 2024, 8:21pm

Hi @jcourt2006 Welcome to the community...

Hi ... Perhaps I am missing something. I do not see an error... you are running with -t which tests the config and exits.

-t, --config.test_and_exit
Check configuration for valid syntax and then exit. Note that grok patterns are not checked for correctness with this flag. Logstash can read multiple config files from a directory. If you combine this flag with --log.level=debug, Logstash will log the combined config file, annotating each config block with the source file it came from.

and that is exactly what it is doing and it is happy

Configuration OK
[2024-01-14T13:56:45,532][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

take out the -t and try again

Also how did you install? tar.gz or a package?

jcourt2006 · January 14, 2024, 8:36pm

Stephend,
Thanks for the quick reaction, in the meantime I found one of the issue in the messages file.
What I found is:

Jan 14 14:30:50 app logstash[77324]: 2024-01-14 14:30:50,796 main ERROR RollingFileManager (/var/log/logstash/logstash-deprecation.log) java.io.FileNotFoundException: /var/log/logstash/logstash-deprecation.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-deprecation.log (Permission denied)
Jan 14 21:11:39 app logstash[85832]: [2024-01-14T15:11:39,223][ERROR][logstash.javapipeline    ][main] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"main", :error=>"(EACCES) Permission denied - /var/log/output-logs-20240114.log", :exception=>Java::OrgJrubyExceptions::SystemCallError, :backtrace=>["org.jruby.RubyIO.sysopen(org/jruby/RubyIO.java:1260)", "org.jruby.RubyFile.initialize(org/jruby/RubyFile.java:364)", "org.jruby.RubyClass.new(org/jruby/RubyClass.java:931)", "org.jruby.RubyIO.new(org/jruby/RubyIO.java:869)", "RUBY.open(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:276)", "usr.share.logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:119)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1601)", "usr.share.logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:118)", "org.jruby.ext.thread.Mutex.synchronize(org/jruby/ext/thread/Mutex.java:171)", "usr.share.logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_output_minus_file_minus_4_dot_3_dot_0.lib.logstash.outputs.file.multi_receive_encoded(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-file-4.3.0/lib/logstash/outputs/file.rb:117)", "usr.share.logstash.logstash_minus_core.lib.logstash.outputs.base.multi_receive(/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:102)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304)"], :thread=>"#<Thread:0x79141328 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 sleep>"}
[root@app logstash]#

There is a Permission denied - /var/log/output-logs-20240114.log.
I solve the issue by changing the directory to /var/log/logstash that is owned by user logstash and group root.

Still Logstash go to shutdown when I send a log to the logfile

I install logstash using Ansible 9.10 on RedHat 9.3 using the Yum module.
I don't expect this kind of issue because this action is the same as performing "yum install logstash" on the server CLI.
To install the repository I got a secure GPG-key file on your forum.

After sending a logger message then the Logstash service shutdown because there is another instance running.


[root@app logstash]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash
Using bundled JDK: /usr/share/logstash/jdk
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2024-01-14T21:47:39,499][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2024-01-14T21:47:39,502][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.11.4", "jruby.version"=>"jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 17.0.9+9 on 17.0.9+9 +indy +jit [x86_64-linux]"}
[2024-01-14T21:47:39,505][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2024-01-14T21:47:39,805][FATAL][logstash.runner          ] Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.
[2024-01-14T21:47:39,810][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:808) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:767) ~[jruby.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
[root@app logstash]#

From the ouput I read: Logstash could not be started because there is already another instance using the configured data directory

i don't know how to solve that.

stephenb · January 14, 2024, 8:52pm

Part of the issue maybe because you're starting logstash from the command line instead of using systemctl which uses the correct permissions and all the correct settings. When you install with a package, the expectation is you'll start with systemctl.

I think you also need to set the --path.config and the --path.data if you want to start from the command line correctly. See here

And then you're not starting actually is the logstash user.

That's why you should start with the systemctl.

And now you've changed some permission so systemctl may or may not work correctly

jcourt2006 · January 14, 2024, 8:59pm

I use the following command as root:
systemctl stop logstash.service
systemctl status logstash.service
systemctl start logstash.service
systemctl restart logstash.service

I will try to do it as user Logstash from CLI and from my Ansible playbook

Also the configuration for path.data is set to "/var/lib/logstash", see below search:
[root@app logstash]# **grep -R path.data ***
logstash.yml:path.data: /var/lib/logstash
logstash.yml:# Default is path.data/queue.
logstash.yml:# Default is path.data/dead_letter_queue
logstash.yml.rpmsave:path.data: /var/lib/logstash
logstash.yml.rpmsave:# Default is path.data/queue.
logstash.yml.rpmsave:# Default is path.data/dead_letter_queue

stephenb · January 14, 2024, 9:12pm

That is probably because you tried initially to start with the wrong user... there is probably to data files / or already a data file with the wrong user ... try cleaning up the data dir.

You should clean up... and run with systemctl commands...

Other thoughts....

For me ... It is much easier to debug logstash from the command line so I get why you are doing that but as we have found out that can mess with the package installation

If you want to test command line commands ... I often recommend having a test box or test directly where you just download and untar the tar.gz distribution... then when you get things the way you want, then copy the pipelines over and use the package installed version..

leandrojmp · January 15, 2024, 1:29am

A couple of things, you should not run logstash a root, this will mess up the permissions of the folders used by logstash.

If you ran logstash as the root user and then try to run it as a service using systemd, it will probably not work because of permissions issues, which seems to be your case.

It will create folders and files inside /var/lib/logstash that are not owned by the logstash user and will also create log files that are not owned by the logstash user, and this will cause the service to not being able to start and sometimes entering a restart loop.

Per default logstash will use the data path as /var/lib/logstash and will write logs in/var/log/logstash/, so both paths needs to be owned by the logstash user and the logstash group.

I would suggest that you stop all running logstash, if any, since it may be on a restart loop, and fix the permissions for both paths.

Also, remove everything inside /var/lib/logstash and /var/log/logstash, then try to run logstash using systemd again or using sudo -u logstash.

Badger · January 15, 2024, 4:00am

Yes, and it sucks that every time I yum update logstash the ownership gets changed to

drwxr-xr-x 2 logstash root 12288 Jan 5 11:41 /var/log/logstash

so log rotation stops until I chgrp it again I always run logstash on the command line as me, not logstash.

jcourt2006 · January 16, 2024, 8:22pm

I want to thank you for the feedback.
I can now install and configure Logstash using Ansible.
Indeed the last issue was because I use root to stop/start/restart logstash, but now I use Logstash with sudo right and it is working.

Thus..... Thank you

Rios · January 16, 2024, 8:56pm

Since is "every time", that means it's the feature

system · February 13, 2024, 8:57pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.