Logstash 6.1.1 do not receive data from Filebeat 6.1.1: ERR Failed to connect

Elastic Stack Logstash
1

I am trying to make filebeat communicate with logstash. It partially works. I have two nodes. First is filebeat-node and this is filebeat.yml:

filebeat.prospectors:
- type: log
enabled: true
paths:
- /home/centos/logs/*.log  
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
output.logstash:
hosts: ["10.206.81.239:5044"]

Second is logstash-node, and logstash.yml looks like this:

path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d/*.conf
path.logs: /var/log/logstash

Logstash pipeline first-pipelien.conf:

input {
    beats {
        port => "5044"
    }
    file{
        path => "/home/centos/logs/mylogs.log"
    }
}
filter {
    grok{
        match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
    }
}
output {
    elasticsearch {
        hosts => [ "10.206.81.246:9200", "10.206.81.236:9200", "10.206.81.243:9200" ]
    }   
     stdout { codec => rubydebug }
}

Now, when I start filebeat by filebeat -e -d "publish and /usr/share/logstash/bin/logstash -f /usr/share/logstash/first-pipeline.conf --config.reload.automatic, everything almost warks. I add log via echo "12.4.14.27 AAA /index.html 138 0.23">> /home/centos/logs/mylogs.log. Filebeat print in console json file with my log and then print:

ERR Failed to publish events caused by: write tcp 10.206.81.235:59922->10.206.81.239:5044: write: connection reset by peer
2018/01/11 08:56:17.774080 output.go:92: ERR Failed to publish events: write tcp 10.206.81.235:59922->10.206.81.239:5044: write: connection reset by peer

Logstash print the same json with log and pass it to elasticsearch (number of doc in index increase). The error looks like logstash restart the connection, so I expect there need to be configured bigger timer for keeping connections (didn't tried yet). But it works in some way.

The problem is, when I start logstash with 'systemctl start logstash'. Logstash is running. When I add some log to filebeat, I receive this errors:

2018/01/11 08:39:32.355756 output.go:74: ERR Failed to connect: dial tcp 10.206.81.239:5044: getsockopt: connection refused
2018/01/11 08:39:34.357051 output.go:74: ERR Failed to connect: dial tcp 10.206.81.239:5044: getsockopt: connection refused
2018/01/11 08:39:38.358227 output.go:74: ERR Failed to connect: dial tcp 10.206.81.239:5044: getsockopt: connection refused
2018/01/11 08:39:46.359342 output.go:74: ERR Failed to connect: dial tcp 10.206.81.239:5044: getsockopt: connection refused

And nothing is sended. Also, on logstash node, there is errors in /var/log/logstash/logstash-plain.log:

[2018-01-11T09:51:28,235][ERROR][io.netty.util.concurrent.DefaultPromise.rejectedExecution] Failed to submit a listener notification task. Event loop shut down?

Output of ls -al /etc/logstash:

drwxrwxr-- 2 logstash logstash   33 10. led 11.37 conf.d
-rwxrwxr-- 1 logstash logstash 1736  9. led 17.34 jvm.options
-rwxrwxr-- 1 logstash logstash 1334 17. pro 22.51 log4j2.properties
-rwxrwxr-- 1 logstash logstash 6454  9. led 16.06 logstash.yml
-rwxrwxr-- 1 logstash logstash 1659 17. pro 22.51 startup.options
drwxr-xr-x 2 root     root       47 11. led 09.52 ${sys:ls.logs}

I set it like this, because I thought, there is problem with permissions for logstash files.

VERSIONS:
logstash 6.1.1
filebeat version 6.1.1 (amd64)

Questions:

  1. Why it works with logstash command but not as a daemon?
  2. How to remove ERR Failed to publish events errors?
  3. How to remove ERR Failed to connect and Failed to submit a listener notification task. Event loop shut down? errors?
  4. What permisions and owners should be set to all files in /etc/logstash directory?
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.