ERR Failed to connect: dial tcp ELK_IP_ADDRESS:9600: getsockopt: connection refused

Elastic Stack Beats
1

For some reason, I cannot get Filebeat to send logs to Logstash. I have an ELK stack on an instance in AWS that is meant to collect all logs from our staging and application servers.

ELK instance:

Elasticsearch -> 5.6.5
Logstash      -> 5.6.5
Kibana        -> 5.6.5

Staging instance:

Filebeat      -> 6.1.2

Filebeat logs from my staging instance:

...
...
2018-03-20T20:05:26Z INFO Harvester started for file: /var/log/ecs/ecs-agent.log.2018-03-20-16
2018-03-20T20:05:27Z ERR  Failed to connect: dial tcp 172.31.1.202:9600: getsockopt: connection refused
2018-03-20T20:05:29Z ERR  Failed to connect: dial tcp 172.31.1.202:9600: getsockopt: connection refused
2018-03-20T20:05:33Z ERR  Failed to connect: dial tcp 172.31.1.202:9600: getsockopt: connection refused
2018-03-20T20:05:41Z ERR  Failed to connect: dial tcp 172.31.1.202:9600: getsockopt: connection refused
2018-03-20T20:05:56Z INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30003 beat.memstats.gc_next=11162992 beat.memstats.memory_alloc=5603560 beat.memstats.memory_total=19392008 filebeat.events.active=4173 filebeat.events.added=4205 filebeat.events.done=32 filebeat.harvester.open_files=39 filebeat.harvester.running=39 filebeat.harvester.started=39 libbeat.config.module.running=0 libbeat.output.type=logstash libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 libbeat.pipeline.events.filtered=78 libbeat.pipeline.events.published=4116 libbeat.pipeline.events.retry=6144 libbeat.pipeline.events.total=4195 registrar.states.current=31 registrar.states.update=32 registrar.writes=32
2018-03-20T20:05:57Z ERR  Failed to connect: dial tcp 172.31.1.202:9600: getsockopt: connection refused
2018-03-20T20:06:26Z INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=11162992 beat.memstats.memory_alloc=6007528 beat.memstats.memory_total=19795976 filebeat.harvester.open_files=39 filebeat.harvester.running=39 libbeat.config.module.running=0 libbeat.config.reloads=1 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 libbeat.pipeline.events.retry=2048 registrar.states.current=31
2018-03-20T20:06:29Z ERR  Failed to connect: dial tcp 172.31.1.202:9600: getsockopt: connection refused
2018-03-20T20:06:56Z INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=11162992 beat.memstats.memory_alloc=6409184 beat.memstats.memory_total=20197632 filebeat.harvester.open_files=39 filebeat.harvester.running=39 libbeat.config.module.running=0 libbeat.config.reloads=1 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 libbeat.pipeline.events.retry=2048 registrar.states.current=31

Filebeat config:

#=============================== Filebeat =====================================
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
    - /var/log/messages
    - /var/log/clamav/*
    - /var/log/ecs/*
    - /var/log/healthchecks/*
  exclude_lines: ['^DBG']

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 30s

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

#================================ Kibana ======================================
#setup.kibana:
#  host: "localhost:5601"

#================================ Outputs =====================================

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["ELK_IP_ADDRESS:9600"]
  
  #protocol: "https"
  #username: "username"
  #password: "password"

  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  #ssl.certificate: "/etc/ssl/filebeat/domain.cert"
  #ssl.key: "/etc/ssl/filebeat/domain.pem"

#================================ Logging =====================================
logging.level: info
#logging.selectors: ["*"]

And finally, my Logstash config on my ELK instance:

http.host: 127.0.0.1 
http.port: 9600-9700 
path.config: /etc/logstash/conf.d 
path.data: /var/lib/logstash 
path.logs: /var/log/logstash 
path.settings: /etc/logstash 
queue.checkpoint.acks: 1024 
queue.checkpoint.interval: 1000 
queue.checkpoint.writes: 1024 
queue.max_bytes: 1024mb 
queue.page_capacity: 250mb 
queue.type: memory

Any guidance would be greatly appreciated.

2

Do you have a beats input configured in your Logstash config? If so, which port is it listening to? This is the port you should use in the filebeat config, not 9600.

3

I do have an additional file with those settings in /etc/logstash/conf.d/ called logstash-syslog.conf:

input {
  file {
    path => ["/var/log/*.log", "/var/log/messages", "/var/log/syslog"]
    type => "syslog"
  }
  beats {
    port => 5140
  }
}

output {
  elasticsearch { 
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri {}
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

Which file is the primary configuration file for Logstash?

  • /etc/logstash/logstash.yml
    or
  • /etc/logstash/conf.d/logstash-syslog.conf?
    The path.config setting, is aiming at /etc/logstash/conf.d, perhapsit isn't being loaded?
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.