Filebeat doesn't send any logs to logstash

[Ferdous_Shibly]

We are getting the following error message in Filebeat

2018-07-19T18:46:18.832Z	ERROR	logstash/async.go:235	Failed to publish events caused by: write tcp 172.17.0.2:42194->10.14.86.242:9191: write: connection reset by peer
2018-07-19T18:46:19.833Z	ERROR	pipeline/output.go:92	Failed to publish events: write tcp 172.17.0.2:42194->10.14.86.242:9191: write: connection reset by peer

But the connectivity is OK.

root@95eff4da9e80:/etc/filebeat# telnet 10.14.86.242 9191
Trying 10.14.86.242...
Connected to 10.14.86.242
Escape character is '^]'.
^]
telnet> quit
Connection closed.

Here is the filebeat configuration

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/src/logs/*.json
  json.message_key: log
  json.keys_under_root: true
  json.add_error_key: true
  multiline.pattern: '^\s'
  multiline.match: after
  document_type: flow-logs
  tail_files: true

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.logstash:
  hosts: ["10.14.86.242:9191"]

And here is the logstash configuration:

input{
  beats {
    host  => "0.0.0.0"
    port  => 9191
    client_inactivity_timeout => 86400
  }
}

output {

    amazon_es {
      hosts  => ["<aws-es>"]
      region => "<aws-region>"
      index  => "flow-logs-%{+YYYY.MM.dd}"
      codec => "json"
    }

}

/usr/share/logstash/bin/logstash-plugin list --verbose beats
logstash-input-beats (5.1.0)

How can I resolve the issue?

[andrewkroh]

Have you tried running Logstash with debug logging enabled to see if there are any clues in the logs there?

Also, what version is your Filebeat?

[Ferdous_Shibly]

We are using filebeat-6.3.1. I don't see any error message on logstash even with debug mode enabled.

output {
   amazon_es {
       hosts  => ["<aws-es>"]
       region => "<aws-region>"
       index  => "flow-logs-%{+YYYY.MM.dd}"
       codec => "json"
   }
   stdout {
      debug => true
   }
}

[andrewkroh]

What does filebeat test output -e -d "*" report?

Another test you could try is to comment out the amazon_es output and see if any events flow. I would enabled the stdout output only.

output { stdout { codec => rubydebug } }

And how did you enable logstash debug? There is a --debug CLI flag or you can enable it through settings?