Filebeat not sending logs to logstash (Versions 5.4.1)

Elastic Stack Logstash
1

I have installed ELK version 5.4.1. Everything went well but for some reason, filebeat cannot send logs to logstash.

2017-06-14T13:59:57+03:00 INFO No non-zero metrics in the last 30s
2017-06-14T14:00:27+03:00 INFO No non-zero metrics in the last 30s
2017-06-14T14:00:46+03:00 ERR Connecting error publishing events (retrying): dial tcp [::1]:5044: getsockopt: connection refused
2017-06-14T14:00:57+03:00 INFO No non-zero metrics in the last 30s
2017-06-14T14:01:27+03:00 INFO No non-zero metrics in the last 30s
2017-06-14T14:01:46+03:00 ERR Connecting error publishing events (retrying): dial tcp [::1]:5044: getsockopt: connection refused
2017-06-14T14:01:57+03:00 INFO No non-zero metrics in the last 30s
2017-06-14T14:02:27+03:00 INFO No non-zero metrics in the last 30s
2017-06-14T14:02:46+03:00 ERR Connecting error publishing events (retrying): dial tcp [::1]:5044: getsockopt: connection refused
2017-06-14T14:02:57+03:00 INFO No non-zero metrics in the last 30s

To add on that, Logstash is not listening on any port. I expected it to listen to port 5044 as configured in filebeat .yml file.

output.logstash:

The Logstash hosts

hosts: ["localhost:5044"]
bulk_max_size: 2048

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

2

One dought

Did you start a logstash to listen on 5044?

Can you put config here ??

3

Here is the log from /var/log/logstash/logstash-plain.log

[2017-06-14T14:53:07,758][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:433:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:216:invalidate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:132:in validate_all'", "org/jruby/RubyHash.java:1342:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:131:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:217:inexecute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:inrun'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in(root)'"]}
[2017-06-14T14:53:37,542][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:433:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:216:invalidate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:132:in validate_all'", "org/jruby/RubyHash.java:1342:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:131:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:217:inexecute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:inrun'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in(root)'"]}

4

As the error message indicates Logstash doesn't start up, but your log snippet is incomplete. In you look in your log, what comes after :error=># on the first line?

Please post your Logstash configuration as requested earlier.

7

log.PNG992×421 21.4 KB

8

@magnusbaeck check below.. I think its a permission issue.. I'm not quite sure . Any heads up ?

log.PNG992×421 21.4 KB

9

Jeez, copy/paste the text instead of posting screenshots.

So is /var/lib/logstash/queue writable to the user running Logstash?

10

@magnusbaeck I took a screenshot because the text after :error=> and before :backtrace=> was getting ommited upon pasting in the editor... Anyhow, Yes /var/lib/logstash/queue is writeable only to the user , in this case, root.

11

@magnusbaeck I finally resolved the issue. My ELK is now working and I can send logs from filebeat and other beats. Thank you.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.