Filebeat hangs when logstash dies

Elastic Stack Beats
1

I'm seeing the connection made to the logstash server stuck in a state of CLOSE_WAIT when logstash goes offline. My expectation is that filebeat would try to re-establish the connection periodically. Anyone else experiencing this? This is version 1.0.0-beta4.

2

can you run filebeat with -d '*,publish' and check output? logstash output has configurable timeout option by default set to 30 seconds.

3

This is what I'm seeing, repeated indefinitely:

prospector.go:130: DBG Start next scan
prospector.go:151: DBG scan path test-*.log
prospector.go:163: DBG Check file for harvesting: test-1.log
prospector.go:280: DBG Update existing file for harvesting: test-1.log
prospector.go:322: DBG Not harvesting, file didn't change: test-1.log
prospector.go:163: DBG Check file for harvesting: test-2.log
prospector.go:280: DBG Update existing file for harvesting: test-2.log
prospector.go:322: DBG Not harvesting, file didn't change: test-2.log
output.go:37: DBG output worker: no events to publish
output.go:37: DBG output worker: no events to publish
output.go:37: DBG output worker: no events to publish
output.go:37: DBG output worker: no events to publish
output.go:37: DBG output worker: no events to publish
output.go:37: DBG output worker: no events to publish
output.go:37: DBG output worker: no events to publish
output.go:37: DBG output worker: no events to publish
output.go:37: DBG output worker: no events to publish
output.go:37: DBG output worker: no events to publish

Should be simple to reproduce: stop & start logstash and you should see filebeat stuck based on output above and lsof for the filebeat process. Also maybe related to this, I noticed during my tests that filebeat was not detecting changes to my test files, even with a very small harvester_buffer_size. Based on the "file didn't change" debug output, perhaps this is the core issue? The CLOSE_WAIT issue might be a separate.

Here is my test filebeat config:

shipper:
  name: test
  geoip:
    paths: []

output:
  logstash:
    enabled: true
    loadbalance: false
    hosts:
      - localhost:12345
    tls:
      disabled: true

filebeat:
  harvester_buffer_size: 128
  prospectors:
    -
      paths:
        - test-*.log
      encoding: utf-8

and logstash config:

input {
  beats {
    type => beats
    port => 12345
  }
}
output {
  stdout {
    codec => rubydebug { metadata => true }
  }
}
4

@erikstephens Have a look at @jhidalgo Currently defining utf-8 is breaking some reading. We are working on a fix for this. In the meantime the only solution is to remove the encoding config.