Lostash becoming unresponsive (?)


(Pantelis Karamolegkos) #1

I am trying to start logstash via

sudo systemctl start logstash

It seems to succeed (service status now is active), however nothing gets written in logstash's own logs, under /var/log/logstash/logstash-plain.log

Until a few minutes ago, starting / stopping the logstash service did produce output in the above log. (e.g. with the respective informative messages on the service having been started or stopped, or the error messages when I messed up come config files).

The system (debian 8) is indeed poor on resources (4GB / 1 Xeon core) however the top command shows log cpu consumption and 1G still available, after starting logstash.

Any suggestions about how can I troubleshoot this?


(Vishal Sharma) #2

well ... do you have enough events generating for logstash to generate the logs ?


(Pantelis Karamolegkos) #3

Well (at least up until a while ago), I did not need any events to see things happening in /var/log/logstash/logstash-plain.log.

Starting and stopping the service was enough to make the file change.

What is more, the client services that are supposed to be sending logs to my logstash instance and on the specific ports I am using, fail to do so (connection refused).


(Pantelis Karamolegkos) #4

dunno if this helps:

● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled)
   Active: active (running) since Fri 2017-09-29 13:55:02 EEST; 51s ago
 Main PID: 112945 (java)
   CGroup: /system.slice/logstash.service
           └─112945 /usr/bin/java -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby....

Sep 29 13:55:02 myhostname systemd[1]: logstash.service holdoff time over, scheduling restart.
Sep 29 13:55:02 myhostname systemd[1]: Stopping logstash...
Sep 29 13:55:02 myhostname systemd[1]: Starting logstash...
Sep 29 13:55:02 myhostname systemd[1]: Started logstash.

It seems to be keeping rescheduling restarts due to some holdoff timer expiring...


(Pantelis Karamolegkos) #5
$ sudo journalctl -f -u logstash
-- Logs begin at Mon 2017-09-18 17:11:57 EEST. --
Sep 29 14:02:21 docker-elk01 systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE
Sep 29 14:02:21 myhostname systemd[1]: Unit logstash.service entered failed state.
Sep 29 14:02:21 myhostname systemd[1]: logstash.service holdoff time over, scheduling restart.
Sep 29 14:02:21 myhostname systemd[1]: Stopping logstash...
Sep 29 14:02:21 myhostname systemd[1]: Starting logstash...
Sep 29 14:02:21 myhostname systemd[1]: Started logstash.
Sep 29 14:03:19 myhostname systemd[1]: Stopping logstash...
Sep 29 14:03:20 myhostname systemd[1]: logstash.service: main process exited, code=exited, status=143/n/a
Sep 29 14:03:20 myhostname systemd[1]: Stopped logstash.
Sep 29 14:03:20 myhostname systemd[1]: Unit logstash.service entered failed state.
Sep 29 14:07:55 myhostname systemd[1]: Starting logstash...
Sep 29 14:07:55 myhostname systemd[1]: Started logstash.
Sep 29 14:09:26 myhostname logstash[115716]: WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Sep 29 14:09:27 myhostname logstash[115716]: Could not find log4j2 configuration at path //etc/logstash/log4j2.properties. Using default config which logs to console
Sep 29 14:09:27 myhostname logstash[115716]: 14:09:27.397 [main] FATAL logstash.runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:433:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:216:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:132:in `validate_all'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:131:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:217:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
Sep 29 14:09:27 myhostname systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE
Sep 29 14:09:27 myhostname systemd[1]: Unit logstash.service entered failed state.
Sep 29 14:09:27 myhostname systemd[1]: logstash.service holdoff time over, scheduling restart.

(Vishal Sharma) #6

[main] FATAL logstash.runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.>

As per your logs make above writable for logstash and restart the service. Let me know if this solved the issue.


(Pantelis Karamolegkos) #7

I did, but now I get this:

Sep 29 14:22:35 myhostname logstash[118552]: WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Sep 29 14:22:35 myhostname logstash[118552]: Could not find log4j2 configuration at path //etc/logstash/log4j2.properties. Using default config which logs to console
Sep 29 14:22:35 myhostname logstash[118552]: ERROR: No configuration file was specified. Perhaps you forgot to provide the '-f yourlogstash.conf' flag?
Sep 29 14:22:35 myhostname logstash[118552]: usage:
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash --help
Sep 29 14:22:36 myhostname systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE

(Vishal Sharma) #8

do you have a conf file in /etc/logstash/conf.d ?
see below from your logs
ERROR: No configuration file was specified. Perhaps you forgot to provide the '-f yourlogstash.conf' flag?


(Pantelis Karamolegkos) #9

Ιt turns out I changed ownership of /etc/logstash by mistake ... giving it to root:root.

This messed up everything.

Apologies for the hassle...


(Vishal Sharma) #10

ah actually i was just checking the log and I missed it as well :frowning:


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.