Hi i'm running into a problem while sending logs via filebeat to logstash:
In short - Can't see logs in kibana - when tailing the filebeat log I see a lot of these:
ERROR logstash/async.go:235 Failed to publish events caused by: read tcp x.x.x.x:36246->y.y,y.y:5045: i/o timeout (while y.y,y.y is logstash address and 5045 is the open beat port)
More details:
I have ~60 machines with filebeat 6.1.1 installed and one logstash machine with logstash 6.2.3 installed.
Some filebeats successfully sends their logs while some throws the error I mentioned above.
those non-errors filebeats sends old logs - means I can see in logstash debug logs that some logs timestamp are 2 or 3 days ago
Logstash usage memory is 35% and cpu usage near 75% on peaks,
in netstat -tupn output in the filebeat machines I can see that the established connections to logstash from filebeat.
Can someone help me find the problem ?