Logstash network output blocking - 7.3.0

Hi. I'm currently having issues with a Logstash instance not being utilized properly. Maximum load is ~ 50% on a 18 core VM running CentOS 7.6.1810. If I replace tcp output{} with file output{} , the load increase to ~90%. (and EPS has a 2x increase)

I have yet to identify the root cause, and not found any evidence pointing towards a unreliable link/host. Destination is able to receive ~800 Mbit/s if i manually netcat data from Logstash host to same destination port, when Logstash in avg transmits 100 Mbit/s. No packets are dropped according to ping, and latency is fine. The destination host is a CentOS 7.6.1810 VM, that receive logs through rsyslog. (imptcp).

Logstash configuration is: (multiple pipelines)
file input pipeline -> filter pipeline -> tcp output pipeline
tcp output pipeline has 6 workers, but only 1-3 are in a running state at any time. the remaining threads are in a interruptable sleeping state (shown as 'S' in htop).

Logstash version: 7.3.0

Trace logs do not reveal any errors.
Moreover, I am not able to identify any TCP resets after analyzing the traffic in Wireshark.

Could anyone please provide suggestions to what I can debug further?

I'll also add that it seems that this blocking have gotten worse after i upgraded to 7.3.0. Earlier, I've managed to reach 70% load on 6.8.2.

Further testing, shows that same issue is apparent on UDP transfers aswell.

Problem caused by overloaded tcp/udp worker. Despite setting up multiple pipeline.workers they still share a single CPU thread responsible for network transmission. Fix was to setup multiple outputs similar to how inputs can be multithreaded.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.