Logstash always restart

Elastic Stack Logstash
1

logstash version:6.0, jdk version: 1.8.0, but no error log for the logstash service, the /var/log/message log is that:
Sep 16 23:17:20 localhost.localdomain systemd[1]: Started logstash.
Sep 16 23:17:30 localhost.localdomain logstash[5740]: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Sep 16 23:17:30 localhost.localdomain systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE
Sep 16 23:17:30 localhost.localdomain systemd[1]: Unit logstash.service entered failed state.
Sep 16 23:17:30 localhost.localdomain systemd[1]: logstash.service failed.
Sep 16 23:17:30 localhost.localdomain systemd[1]: logstash.service holdoff time over, scheduling restart.
Sep 16 23:17:30 localhost.localdomain systemd[1]: Stopped logstash.
Sep 16 23:17:30 localhost.localdomain systemd[1]: Started logstash.

2

Please provide log sample of /var/log/logstash/logstash-plain.log

3

This is the log:
[2020-09-16T23:37:18,317][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2020-09-16T23:37:18,320][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2020-09-16T23:37:18,340][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:443:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:225:in validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:136:in block in validate_all'", "org/jruby/RubyHash.java:1343:in each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:135:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:280:in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:232:in run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in '"]}

4

<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable

You have some permissions error on your configuration.

5

I have change the owner for the every files in the /var/lib/logstash, but still have the error:
[2020-09-16T23:54:14,165][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<Errno::EADDRNOTAVAIL: Cannot assign requested address - bind - Cannot assign requested address>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:127:in initialize'", "org/jruby/RubyIO.java:875:in new'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/binder.rb:234:in add_tcp_listener'", "(eval):2:in add_tcp_listener'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:88:in start_webserver'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:44:in block in run'", "org/jruby/RubyRange.java:485:in each'", "org/jruby/RubyEnumerable.java:1067:in each_with_index'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:39:in run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:394:in block in start_webserver'"]}

6

What's your logstash configuration ?

It seems that your network address is not avaibable probably because the process is stalled or you missconfigured and IP address.

Please ensure logstash is killed cleanly and restart.

7

My logstash config file is:
input {
syslog {
type => "system-syslog"
port => 10514
}
}
output {
elasticsearch {
hosts => ["192.168.56.21:9200"]
index => "system.sys.%{+YYY.MM}"
}
}
The 192.168.56.21 is the elasticsearch master, and I can telnet the port 9200 from logstash vm.
[root@localhost conf.d]# telnet 192.168.56.21 9200
Trying 192.168.56.21...
Connected to 192.168.56.21.
Escape character is '^]'.

8

logstash is trying to start the webserver used for monitoring it. What is http.host set to in logstash.yml? That is the IP it will be failing to bind to.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.