pkaramol
September 29, 2017, 10:26am
1
I am trying to start logstash via
sudo systemctl start logstash
It seems to succeed (service status now is active), however nothing gets written in logstash's own logs, under /var/log/logstash/logstash-plain.log
Until a few minutes ago, starting / stopping the logstash service did produce output in the above log. (e.g. with the respective informative messages on the service having been started or stopped, or the error messages when I messed up come config files).
The system (debian 8) is indeed poor on resources (4GB / 1 Xeon core) however the top command shows log cpu consumption and 1G still available, after starting logstash.
Any suggestions about how can I troubleshoot this?
well ... do you have enough events generating for logstash to generate the logs ?
pkaramol
September 29, 2017, 10:35am
3
Well (at least up until a while ago), I did not need any events to see things happening in /var/log/logstash/logstash-plain.log.
Starting and stopping the service was enough to make the file change.
What is more, the client services that are supposed to be sending logs to my logstash instance and on the specific ports I am using, fail to do so (connection refused).
pkaramol
September 29, 2017, 10:57am
4
dunno if this helps:
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; disabled)
Active: active (running) since Fri 2017-09-29 13:55:02 EEST; 51s ago
Main PID: 112945 (java)
CGroup: /system.slice/logstash.service
└─112945 /usr/bin/java -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby....
Sep 29 13:55:02 myhostname systemd[1]: logstash.service holdoff time over, scheduling restart.
Sep 29 13:55:02 myhostname systemd[1]: Stopping logstash...
Sep 29 13:55:02 myhostname systemd[1]: Starting logstash...
Sep 29 13:55:02 myhostname systemd[1]: Started logstash.
It seems to be keeping rescheduling restarts due to some holdoff timer expiring...
pkaramol
September 29, 2017, 11:19am
5
$ sudo journalctl -f -u logstash
-- Logs begin at Mon 2017-09-18 17:11:57 EEST. --
Sep 29 14:02:21 docker-elk01 systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE
Sep 29 14:02:21 myhostname systemd[1]: Unit logstash.service entered failed state.
Sep 29 14:02:21 myhostname systemd[1]: logstash.service holdoff time over, scheduling restart.
Sep 29 14:02:21 myhostname systemd[1]: Stopping logstash...
Sep 29 14:02:21 myhostname systemd[1]: Starting logstash...
Sep 29 14:02:21 myhostname systemd[1]: Started logstash.
Sep 29 14:03:19 myhostname systemd[1]: Stopping logstash...
Sep 29 14:03:20 myhostname systemd[1]: logstash.service: main process exited, code=exited, status=143/n/a
Sep 29 14:03:20 myhostname systemd[1]: Stopped logstash.
Sep 29 14:03:20 myhostname systemd[1]: Unit logstash.service entered failed state.
Sep 29 14:07:55 myhostname systemd[1]: Starting logstash...
Sep 29 14:07:55 myhostname systemd[1]: Started logstash.
Sep 29 14:09:26 myhostname logstash[115716]: WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Sep 29 14:09:27 myhostname logstash[115716]: Could not find log4j2 configuration at path //etc/logstash/log4j2.properties. Using default config which logs to console
Sep 29 14:09:27 myhostname logstash[115716]: 14:09:27.397 [main] FATAL logstash.runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:433:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:216:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:132:in `validate_all'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:131:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:217:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
Sep 29 14:09:27 myhostname systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE
Sep 29 14:09:27 myhostname systemd[1]: Unit logstash.service entered failed state.
Sep 29 14:09:27 myhostname systemd[1]: logstash.service holdoff time over, scheduling restart.
[main] FATAL logstash.runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.>
As per your logs make above writable for logstash and restart the service. Let me know if this solved the issue.
pkaramol
September 29, 2017, 11:23am
7
I did, but now I get this:
Sep 29 14:22:35 myhostname logstash[118552]: WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Sep 29 14:22:35 myhostname logstash[118552]: Could not find log4j2 configuration at path //etc/logstash/log4j2.properties. Using default config which logs to console
Sep 29 14:22:35 myhostname logstash[118552]: ERROR: No configuration file was specified. Perhaps you forgot to provide the '-f yourlogstash.conf' flag?
Sep 29 14:22:35 myhostname logstash[118552]: usage:
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
Sep 29 14:22:35 myhostname logstash[118552]: bin/logstash --help
Sep 29 14:22:36 myhostname systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE
do you have a conf file in /etc/logstash/conf.d ?
pkaramol
September 29, 2017, 11:52am
9
Ιt turns out I changed ownership of /etc/logstash by mistake ... giving it to root:root.
This messed up everything.
Apologies for the hassle...
ah actually i was just checking the log and I missed it as well
system
October 27, 2017, 12:05pm
11
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
