Logstash with Filebeat i/o timeout

[mschuster (Martin)]

Hi all,

i got some problems with filebeat and logstash. When i start logstash, all informations are going into elasticsearch, but after a short time i don´t see (most) informations from filebeat in kibana.
In the logfiles (elastic/logstash) there are no errros, the only information i see on the clients (filebeat) is

2018-02-22T14:10:56.886+0100 ERROR logstash/async.go:235 Failed to publish events caused by: read tcp x.x.x.x:58978->x.x.x.x:5044: i/o timeout
2018-02-22T14:10:56.886+0100 ERROR logstash/async.go:235 Failed to publish events caused by: read tcp x.x.x.x:58978->x.x.x.x:5044: i/o timeout
2018-02-22T14:10:56.886+0100 ERROR logstash/async.go:235 Failed to publish events caused by: read tcp x.x.x.x:58978->x.x.x.x:5044: i/o timeout
2018-02-22T14:10:56.887+0100 ERROR logstash/async.go:235 Failed to publish events caused by: client is not connected
2018-02-22T14:10:57.887+0100 ERROR pipeline/output.go:92 Failed to publish events: client is not connected

We are running:

3x logstash (6.2.2)

• logstash-codec-beats 5.0.10
  3x elasticsearch (6.2.2)
  and for debugging just 160 filebeats (6.2.2)

Thx and regards,
Martin

[alesf]

Hi Martin,

I just joined the club and have same issue here: after a while (good running),
there are ERRORs showing up and system (kibana) does not show logs anymore...
And can't figure yet, why/how to restore this back to start working again.

Do you have any new info on this?

Thanks
Ales

2018-03-21T09:16:26.540+0100 INFO crawler/crawler.go:48 Loading Prospectors: 3
2018-03-21T09:16:26.541+0100 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-03-21T09:16:26.542+0100 INFO log/prospector.go:111 Configured paths: [/var/opt/tomcats/fep/logs/bridge-chi-transport.log /var/opt/tomcats/fep/logs/chocolate-starfish.log]
2018-03-21T09:16:26.543+0100 INFO log/prospector.go:111 Configured paths: [/var/log/logstash/logstash-plain*.log]
2018-03-21T09:16:26.543+0100 INFO log/prospector.go:111 Configured paths: [/var/log/logstash/logstash-slowlog-plain*.log]
2018-03-21T09:16:26.543+0100 INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 3
2018-03-21T09:16:26.544+0100 INFO log/harvester.go:216 Harvester started for file: /var/opt/tomcats/fep/logs/chocolate-starfish.log
2018-03-21T09:16:26.544+0100 INFO log/harvester.go:216 Harvester started for file: /var/opt/tomcats/fep/logs/bridge-chi-transport.log
2018-03-21T09:16:56.543+0100 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":30,"time":36},"total":{"ticks":290,"time":296,"value":290},"user":{"ticks":260,"time":260}},"info":{"ephemeral_id":"dfe46419-7361-4187-8b31-83918f3fcfb0","uptime":{"ms":30017}},"memstats":{"gc_next":15944048,"memory_alloc":9971264,"memory_total":36771832,"rss":25247744}},"filebeat":{"events":{"active":4118,"added":4122,"done":4},"harvester":{"open_files":2,"running":2,"started":2}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":4096,"batches":2,"total":4096},"type":"logstash","write":{"bytes":273720}},"pipeline":{"clients":3,"events":{"active":4117,"filtered":4,"published":4116,"retry":2048,"total":4121}}},"registrar":{"states":{"current":20,"update":4},"writes":4},"system":{"cpu":{"cores":1},"load":{"1":0,"15":0.05,"5":0.01,"norm":{"1":0,"15":0.05,"5":0.01}}}}}}
2018-03-21T09:16:56.765+0100 ERROR logstash/async.go:235 Failed to publish events caused by: read tcp 10.19.x.x:38910->10.17.x.x:5044: i/o timeout
2018-03-21T09:16:56.765+0100 ERROR logstash/async.go:235 Failed to publish events caused by: read tcp 10.19.x.x:38910->10.17.x.x:5044: i/o timeout
2018-03-21T09:16:56.826+0100 ERROR logstash/async.go:235 Failed to publish events caused by: client is not connected
2018-03-21T09:16:57.826+0100 ERROR pipeline/output.go:92 Failed to publish events: client is not connected