Hi everyone,
I will review the 2 links provided by Tudor.
Robert, the HDD architecture is handled by another team. It think it's SAS HDD, and probably not the fastest architecture ever, however i was dropping a ton of packets on Logstash with 16 vCPU and now with 32 vCPU i didn't drop a single packet out of 20.000.000. Elasticsearch is composed of 3 nodes and right now doesn't look pressured, so i'm pretty sure that my initial problem here wasn't about backpressure.
However, if i continue to add more and more input into my cluster, it might become the case.
On the Logstash perspective, i receive Netflow trafic from only one router, so i can't loadbalance directly onto it.
I'm waiting the next week to see how it behave on a busy day (it's Easter Holliday right now, so there isn't that much flows).
If i start dropping again, maybe i should think about splitting my Logstash into 2 instances with 16 cores or 4 instances with 8 cores, with a load balancer... But it really looks like "too much" for me...
We had an old and shitty Collector for years, and it could handle everything with only 8 CPU. Maybe Logstash add more info' (like GeoIP), but 8 CPU vs 32 CPU is a lot more ressources.
Thank you for you advices.