hi @tudor,
It's truly multiple filebeats -> cluster of logstashes -> ES.
All filebeat nodes talk to the same set of logstashes (with loadbalance: true and publish_async: true). I have logstash,output.timeout in filebeat increased to 300s and congestion_threshold set to 90 sec in beats input on LS side.
Unfortunately logs itself very already rotated out, but I'll try to catch this problem tomorrow. But from what I was able to check when I spotted this issue there was completely no registrar logs -- so this is very similar to what I've reported in Filebeat beta1 resends random data upon every restart (registry file not updated properly?)
One ultra important thing however: as being said, it only happens on some (usually single) filebeat nodes and we're talking about entire data set being replayed (it's always full set) -- in case of today that was data from last 8hrs worth 30M of events, so I just have virtually no capacity to "buffer" that amount of events on logstash side without acking.
Also before filebeat node was restarted all events coming from it were processed and stored in ES without any lag (in less than 2s).