Logstash network output blocking - 7.3.0

[espenk (neeps)]

Hi. I'm currently having issues with a Logstash instance not being utilized properly. Maximum load is ~ 50% on a 18 core VM running CentOS 7.6.1810. If I replace tcp output{} with file output{} , the load increase to ~90%. (and EPS has a 2x increase)

I have yet to identify the root cause, and not found any evidence pointing towards a unreliable link/host. Destination is able to receive ~800 Mbit/s if i manually netcat data from Logstash host to same destination port, when Logstash in avg transmits 100 Mbit/s. No packets are dropped according to ping, and latency is fine. The destination host is a CentOS 7.6.1810 VM, that receive logs through rsyslog. (imptcp).

Logstash configuration is: (multiple pipelines)
file input pipeline -> filter pipeline -> tcp output pipeline
tcp output pipeline has 6 workers, but only 1-3 are in a running state at any time. the remaining threads are in a interruptable sleeping state (shown as 'S' in htop).

Logstash version: 7.3.0

Trace logs do not reveal any errors.
Moreover, I am not able to identify any TCP resets after analyzing the traffic in Wireshark.

Could anyone please provide suggestions to what I can debug further?

[espenk (neeps)]

I'll also add that it seems that this blocking have gotten worse after i upgraded to 7.3.0. Earlier, I've managed to reach 70% load on 6.8.2.

[espenk (neeps)]

Further testing, shows that same issue is apparent on UDP transfers aswell.

[espenk (neeps)]

Problem caused by overloaded tcp/udp worker. Despite setting up multiple pipeline.workers they still share a single CPU thread responsible for network transmission. Fix was to setup multiple outputs similar to how inputs can be multithreaded.