Can anyone share experience about server scaling and UDP performance.
my Logstash setup is quite simple:
UDP input with Netflow Codec -> no Filters -> Elasticsearch output via HTTP via bulk API
My current test environment is based on a quite outdated 4Core Xeon (E5320) with 16GB RAM and 10k SAS drives in Raid1 configuration.
I'm collecting about 2k flows per second from one of our edge routers which causes an average system load of 4.0 during daily peaks.
Nearly all of the CPU load ist caused by the Java process Logstash runs in. ElasticSearch only utilizes half of an core in average.
I'm wondering if this is a normal behavior? I noticed that Logstash itself is eligible to process 100k of events per second, so i'm wondering that 2% of this are causing such a high load.
Is JSON serialisation for ES output causing this high load?
Our production environment currently produces daily peaks of 10k flows/sec. My production hardware should be able to process 20k flows/sec. What will be a suitable server configuration here?
Thanks in advance for all replys