Filebeat: Connection to Logstash fails, then immediately tries backoff and that works

Elastic Stack Beats
1

Hi guys,

I have a dockerized Filebeat 6.5.1 instance that monitors my docker containers using the docker input.

Basically every time a log line is found by Filebeat, I get a Failed to publish events / connection reset by peer", but then it apparently connects again immediately, and then it works, so it cannot really be a firewall issue, can it?

(Also, pinging the Logstash server from inside the Filebeat container works fine, and netstat tells me that Logstash does listen on port 5045. Firewall is open - also, in the 2nd try it succeeds, to it cannot be the firewall, I guess).

{"level":"info","timestamp":"2018-11-30T14:01:27.096Z","caller":"log/harvester.go:254","message":"Harvester started for file: /var/lib/docker/containers/8d94925c6fe48467db3d/8d94925c6fe48467db3df-json.log"}
{"level":"error","timestamp":"2018-11-30T14:01:28.106Z","caller":"logstash/async.go:256","message":"Failed to publish events caused by: write tcp 172.22.0.2:44042->123.456.12.14:5045: write: connection reset by peer"}
{"level":"error","timestamp":"2018-11-30T14:01:29.107Z","caller":"pipeline/output.go:121","message":"Failed to publish events: write tcp 172.22.0.2:44042->123.456.12.14:5045: write: connection reset by peer"}
{"level":"info","timestamp":"2018-11-30T14:01:29.108Z","caller":"pipeline/output.go:95","message":"Connecting to backoff(async(tcp://foo.bar.de:5045))"}
{"level":"info","timestamp":"2018-11-30T14:01:29.119Z","caller":"pipeline/output.go:105","message":"Connection to backoff(async(tcp://foo.bar.de:5045)) established"}

Interesting, for the first try it prints the IP of the logstash server, the second time it prints the host name - could that have to do with IPv4/IPv6 issues?

I find this weird and would be grateful about any hints! As far as I can see the logs always get sent in the 2nd try, but this behaviour is probably not how it should be, right?

Thanks!

__
__
__

This is my docker-compose:

version: "3.3"
services:

  filebeat:
    user: root
    image: docker.elastic.co/beats/filebeat:6.5.1
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/:/var/lib/docker:ro
      - ./logs/:/usr/share/filebeat/logs/
    entrypoint: "/usr/local/bin/docker-entrypoint" # to enable logging to file
    command: "-E FOO=BAR" # to overwrite the -e that disables logging to file!

This is my filebeat.yml:

filebeat.inputs:
  - type: docker
    containers.ids: '*'
    ignore_older: 1h
    enabled: true
    processors:
    - add_docker_metadata: ~

output:
  logstash:
    hosts: ["foo.bar.de:5045"]
    timeout: 240s
    enabled: true
  console:
    enabled: false # otherwise they get printed to stdout and get picked up via docker again - eternal circle!
2

Also check logstash settings and logs. Logstash also times out idle connections.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.