Filebeat shows i/o timeout when pushing file

Elastic Stack Beats
1

Hi,

I have a very simple setup. I have the ELK stack in a docker container. I am using the latest image from here https://hub.docker.com/r/sebp/elk/.

I've installed Filebeat (6.6.1) on the machine. The configuration file that I am using for Filebeat is this one:

output:
  logstash:
    enabled: true
    hosts:
      - localhost:5044
    ssl:
      certificate_authorities:
        - /etc/pki/tls/certs/logstash-beats.crt
    timeout: 15

filebeat.prospectors:
- type: log
  paths:
    - /tmp/*.json
  json.keys_under_root: true
  json.overwrite_keys: true

setup.template.name: "mylogs"
setup.template.pattern: "mylogs"

I have put the .crt file in this path "/etc/pki/tls/certs/logstash-beats.crt".

After I start the whole environment and I put a "json" file in /tmp folder everything works fine. The logs arrive in log stash and I can see them in Kibana. The problem appears if after a few minutes, I add a second file in that folder. I start seeing this.

2019-03-05T10:51:26.163-0800	INFO	log/harvester.go:255	Harvester started for file: /tmp/qqq1.json
2019-03-05T10:51:26.279-0800	ERROR	logstash/async.go:256	Failed to publish events caused by: EOF
2019-03-05T10:51:26.318-0800	ERROR	logstash/async.go:256	Failed to publish events caused by: client is not connected
2019-03-05T10:51:28.127-0800	ERROR	pipeline/output.go:121	Failed to publish events: client is not connected
2019-03-05T10:51:28.127-0800	INFO	pipeline/output.go:95	Connecting to backoff(async(tcp://localhost:5044))
2019-03-05T10:51:35.924-0800	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":910,"time":{"ms":64}},"total":{"ticks":6556,"time":{"ms":346},"value":6556},"user":{"ticks":5646,"time":{"ms":282}}},"info":{"ephemeral_id":"cbe29447-e7c6-491a-9dca-5ec204b49de0","uptime":{"ms":270201}},"memstats":{"gc_next":23260592,"memory_alloc":13159008,"memory_total":650281088,"rss":20602880}},"filebeat":{"events":{"active":4117,"added":4118,"done":1},"harvester":{"open_files":3,"running":3,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"batches":2,"failed":4096,"total":4096},"read":{"bytes":31},"write":{"bytes":173242}},"pipeline":{"clients":1,"events":{"active":4117,"filtered":1,"published":4116,"retry":2048,"total":4118}}},"registrar":{"states":{"current":3,"update":1},"writes":{"success":1,"total":1}},"system":{"load":{"1":1.9077,"15":1.7837,"5":2.021,"norm":{"1":0.4769,"15":0.4459,"5":0.5052}}}}}}
2019-03-05T10:51:46.554-0800	ERROR	pipeline/output.go:100	Failed to connect to backoff(async(tcp://localhost:5044)): read tcp [::1]:50768->[::1]:5044: i/o timeout
2019-03-05T10:51:46.564-0800	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 1 reconnect attempt(s)
2019-03-05T10:52:05.925-0800	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":914,"time":{"ms":4}},"total":{"ticks":6562,"time":{"ms":6},"value":6562},"user":{"ticks":5648,"time":{"ms":2}}},"info":{"ephemeral_id":"cbe29447-e7c6-491a-9dca-5ec204b49de0","uptime":{"ms":300201}},"memstats":{"gc_next":23260592,"memory_alloc":13779312,"memory_total":650901392,"rss":544768}},"filebeat":{"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"errors":2},"write":{"bytes":160}},"pipeline":{"clients":1,"events":{"active":4117,"retry":2048}}},"registrar":{"states":{"current":3}},"system":{"load":{"1":1.7744,"15":1.7778,"5":1.9839,"norm":{"1":0.4436,"15":0.4445,"5":0.496}}}}}}
2019-03-05T10:52:06.989-0800	ERROR	pipeline/output.go:100	Failed to connect to backoff(async(tcp://localhost:5044)): read tcp 127.0.0.1:50771->127.0.0.1:5044: i/o timeout
2019-03-05T10:52:06.989-0800	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 2 reconnect attempt(s)

After this happens, if I try to access Kibana from browser it won't work anymore. On the elk stack side, I don't see any relevant logs.

I've tried to telnet into localhost 5044 and I manage to connect successfully.

For debugging purposes, I've switched the input in filebeat to "stdin" and it is working well.

I am out of ideas what I could try. Any help is welcomed.

LE: I've hosted the ELK stack on a different machine without Docker and it seems like it is working fine.

Thanks,
Bogdan

2

This is weird, I haven't used that specific docker images, but I would check Logstash's logs to see if there is any errors there.

FYI we also have our official dockers images available at https://www.docker.elastic.co/

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.