Hi,
I have a very simple setup. I have the ELK stack in a docker container. I am using the latest image from here https://hub.docker.com/r/sebp/elk/.
I've installed Filebeat (6.6.1) on the machine. The configuration file that I am using for Filebeat is this one:
output:
logstash:
enabled: true
hosts:
- localhost:5044
ssl:
certificate_authorities:
- /etc/pki/tls/certs/logstash-beats.crt
timeout: 15
filebeat.prospectors:
- type: log
paths:
- /tmp/*.json
json.keys_under_root: true
json.overwrite_keys: true
setup.template.name: "mylogs"
setup.template.pattern: "mylogs"
I have put the .crt file in this path "/etc/pki/tls/certs/logstash-beats.crt".
After I start the whole environment and I put a "json" file in /tmp folder everything works fine. The logs arrive in log stash and I can see them in Kibana. The problem appears if after a few minutes, I add a second file in that folder. I start seeing this.
2019-03-05T10:51:26.163-0800 INFO log/harvester.go:255 Harvester started for file: /tmp/qqq1.json
2019-03-05T10:51:26.279-0800 ERROR logstash/async.go:256 Failed to publish events caused by: EOF
2019-03-05T10:51:26.318-0800 ERROR logstash/async.go:256 Failed to publish events caused by: client is not connected
2019-03-05T10:51:28.127-0800 ERROR pipeline/output.go:121 Failed to publish events: client is not connected
2019-03-05T10:51:28.127-0800 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://localhost:5044))
2019-03-05T10:51:35.924-0800 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":910,"time":{"ms":64}},"total":{"ticks":6556,"time":{"ms":346},"value":6556},"user":{"ticks":5646,"time":{"ms":282}}},"info":{"ephemeral_id":"cbe29447-e7c6-491a-9dca-5ec204b49de0","uptime":{"ms":270201}},"memstats":{"gc_next":23260592,"memory_alloc":13159008,"memory_total":650281088,"rss":20602880}},"filebeat":{"events":{"active":4117,"added":4118,"done":1},"harvester":{"open_files":3,"running":3,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"batches":2,"failed":4096,"total":4096},"read":{"bytes":31},"write":{"bytes":173242}},"pipeline":{"clients":1,"events":{"active":4117,"filtered":1,"published":4116,"retry":2048,"total":4118}}},"registrar":{"states":{"current":3,"update":1},"writes":{"success":1,"total":1}},"system":{"load":{"1":1.9077,"15":1.7837,"5":2.021,"norm":{"1":0.4769,"15":0.4459,"5":0.5052}}}}}}
2019-03-05T10:51:46.554-0800 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://localhost:5044)): read tcp [::1]:50768->[::1]:5044: i/o timeout
2019-03-05T10:51:46.564-0800 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 1 reconnect attempt(s)
2019-03-05T10:52:05.925-0800 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":914,"time":{"ms":4}},"total":{"ticks":6562,"time":{"ms":6},"value":6562},"user":{"ticks":5648,"time":{"ms":2}}},"info":{"ephemeral_id":"cbe29447-e7c6-491a-9dca-5ec204b49de0","uptime":{"ms":300201}},"memstats":{"gc_next":23260592,"memory_alloc":13779312,"memory_total":650901392,"rss":544768}},"filebeat":{"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"errors":2},"write":{"bytes":160}},"pipeline":{"clients":1,"events":{"active":4117,"retry":2048}}},"registrar":{"states":{"current":3}},"system":{"load":{"1":1.7744,"15":1.7778,"5":1.9839,"norm":{"1":0.4436,"15":0.4445,"5":0.496}}}}}}
2019-03-05T10:52:06.989-0800 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://localhost:5044)): read tcp 127.0.0.1:50771->127.0.0.1:5044: i/o timeout
2019-03-05T10:52:06.989-0800 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 2 reconnect attempt(s)
After this happens, if I try to access Kibana from browser it won't work anymore. On the elk stack side, I don't see any relevant logs.
I've tried to telnet into localhost 5044 and I manage to connect successfully.
For debugging purposes, I've switched the input in filebeat to "stdin" and it is working well.
I am out of ideas what I could try. Any help is welcomed.
LE: I've hosted the ELK stack on a different machine without Docker and it seems like it is working fine.
Thanks,
Bogdan