We are receiving BIG IP F5 request-logs in Logstash on a tcp-input (with codec => "json_lines") in our production environment. We see that a lot of messages (json) are truncated and result in many (thousands per hour) jsonparsefailures or are missing. We cannot reproduce these jsonparsefailures and missing messages in our test environment.
When we capture the traffic between F5 and logstash we see that Logstash sends a "Zero Window" probe meaning it's not able to deal with the traffic/data sent and is telling the F5 to pause until Logstash has freed its buffer. We can increase the Zero window timeout setting on the F5 but we really want to solve this in Logstash. We are running Logstash on Kubernetes/Docker and we tried increasing the number of Logstash pods or increase kernel parameters like 'net.ipv4.tcp_rmem' and 'net.ipv4.tcp_wmem' but all without any results.
My question is: how can we tune Logstash to handle the incoming load properly.