Filebeat "TCP connection refused" error on DCOS cluster

[aysala]

Hello,

I am trying to run Filebeat on all the nodes(Master, Agent) in my DCOS cluster. Filebeat works fine with ES as output in the configuration. But I get the 'dial tcp : getsockopt connection refused' error when logstash is used as the output.

Elastic & Kibana version- 5.6.5
Logstash version - 5.6-alpine
Filebeat version - 6.2.2

Error:

# tail -f /var/log/filebeat/filebeat
2018-02-22T21:13:44.972Z        WARN    [cfgwarn]       prospector/config.go:25 DEPRECATED: input_type prospector config is deprecated. Use type instead. Will be removed in version: 6.0.0
2018-02-22T21:13:44.973Z        INFO    log/prospector.go:111   Configured paths: [/var/log/yum.log]
2018-02-22T21:13:44.973Z        INFO    crawler/crawler.go:82   Loading and starting Prospectors completed. Enabled prospectors: 1
2018-02-22T21:13:44.974Z        INFO    log/harvester.go:216    Harvester started for file: /var/log/yum.log
2018-02-22T21:13:46.996Z        ERROR   pipeline/output.go:74   Failed to connect: dial tcp xxxxxxxxxxx:5044: getsockopt: connection refused
2018-02-22T21:13:49.003Z        ERROR   pipeline/output.go:74   Failed to connect: dial tcp xxxxxxxxxxx:5044: getsockopt: connection refused
2018-02-22T21:13:53.009Z        ERROR   pipeline/output.go:74   Failed to connect: dial tcp xxxxxxxxxxx:5044: getsockopt: connection refused
2018-02-22T21:14:01.015Z        ERROR   pipeline/output.go:74   Failed to connect: dial tcp xxxxxxxxxxx:5044: getsockopt: connection refused
2018-02-22T21:14:17.037Z        ERROR   pipeline/output.go:74   Failed to connect: dial tcp xxxxxxxxxxx:5044: getsockopt: connection refused

filebeat configuration:

filebeat.prospectors:
- input_type: log
  paths:
  - /var/log/yum.log
#  - /var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/stdout
#  - /var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/stderr
#  - /var/log/mesos/*.log
#  - /var/log/dcos/dcos.log
exclude_files: ["stdout.logrotate.state", "stdout.logrotate.conf", "stderr.logrotate.state", "stderr.logrotate.conf"]
tail_files: true
#output.elasticsearch:
#   hosts: ["http://coordinator.elastic.l4lb.thisdcos.directory:9200"]
output.logstash:
   hosts: ["logstash.marathon.mesos:5044"]
   bulk_max_size: 1024

Logstash configuration:

        input { 
          beats { port=> 5044 }
         }
        output { 
        elasticsearch { 
        hosts => 'elasticsearch-executor.elasticsearch.mesos:1025' 
        index => '%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}  
         }
        }

Telnet is giving the same connection refused error. Also I made sure that the port is open. Running out of ideas here.

[andrewkroh]

@aysala

Telnet is giving the same connection refused error.

Then I would first check the LS container by doing a netstat -an | grep 5044 inside the container to make sure it's listening as expected.

Then make sure that 5044 is exposed from the container and that there is a port mapping.

I don't have any DC/OS experience so I can't help you dig in much further.