Error After updating to ELK 5

Elastic Stack Logstash
1

Hi,
I did a upgrade to ELK 5 but all of the sudden it stopped working

I checked my logs and it says

ERR Connecting error publishing events (retrying): dial tcp 192.168.154.100:5044: getsockopt: connection refused

My config for beats is like this:

input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

#pfSense
#tcp syslog stream via 5140
input {
  tcp {
    type => "syslog"
    port => 5140
  }
}
#udp syslogs tream via 5140
input {
  udp {
    type => "syslog"
    port => 5140
  }
}

and my filebeat.yml looks like this

filebeat.prospectors:

  • input_type: log
    paths:
    • /var/log/secure
    • /var/log/messages
      document_type: syslog
  • input_type: log
    paths:
    • /var/log/yum.log
      document_type: yumlog
  • input_type: log
    paths:
    • /var/log/nginx-access
      document_type: nginx-access
  • input_type: log
    paths:
    • /var/log/httpd/apache-access
      document_type: apache-access
  • input_type: log
    paths:
    • /var/log/cronlog
      document_type: cronlog
  • input_type: log
    paths:
    • /var/log/auditlog
      document_type: auditlog
  • input_type: log
    paths:
    • /var/log/maillog
      document_type: maillog

output.logstash:
hosts: ["192.168.154.100:5044"]

ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

It was working fine before but now it doesn't and I just don't get why?

2

Does Logstash start up properly? Have you looked in the Logstash log (typically in /var/log/logstash)?

3

HI @magnusbaeck thanks for the reply
Yes it does
When I type
sudo systemctl status logstash

it gives me

logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2017-05-10 16:23:36 CEST; 1s ago
Main PID: 14345 (java)
CGroup: /system.slice/logstash.service
└─14345 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatin...

May 10 16:23:36 kibana systemd[1]: logstash.service holdoff time over, scheduling restart.
May 10 16:23:36 kibana systemd[1]: Started logstash.
May 10 16:23:36 kibana systemd[1]: Starting logstash...

And I checked the logs but

logstash.err is empty
logstash.log it says

{:timestamp=>"2017-05-10T15:23:02.163000+0200", :message=>"Cannot get new connection from pool.", :class=>"Elasticsearch::Transport::Transport::Error", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:193:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/manticore.rb:54:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/client.rb:125:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.15/lib/elasticsearch/api/actions/bulk.rb:87:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk

And it keeps on going like this I don't understand it

I really don't know what to do