Understanding event loss with HTTP input and jmeter test

Elastic Stack Logstash
1

Hi,

I'm trying to configure & stress test an HTTP input agent.
The agent for the sake of example cant be as simple as HTTP input and not much more.
At around 9000 messages per second, I start seeing events that time out. The number grows little by little.
Where is my bottleneck? Is it CPU? RAM? I'm not seeing any major jumps in RAM usage when sending events (which is also a bit suspicous) .
My goal is to reach an ingest rate of around 10k events with no loss.

Thanks,

2

Are you using bulk requests? How many parallel indexing threads are you using? Are you monitoring disk IO utilisation, which can often be a limiting factor?

3

I am not using bulk requests. Due to the nature of the system, I need to get an event and process it immediately.
I'm using 8 indexing workers.
Disk IO isn't an issue.
My hunch is that it is a TCP issue - I'm just unsure if it's the HTTP plugin or the server itself.