Hi All,
I am currently in a Docker swarm setup with 7 nodes ingesting 20K 200 bytes TPS .
Recv-Q is at almost 4.000.000 segments waiting to be processed by logstash
right now I have 80 workers, batch_size=1000 and pool_max=5000.
Have been tweaking server's TCP buffers but there is no much difference.
Read backpressure can go as high as 50, output write backpressure is low 2-4.
Any ideas to accelerate ingestion so Recv-Q is lowered?
Thank you very much
Daniel