Hi i tested ELK stack to use in our enviroment. I send about 5k and more for one logstash node but always saw in Kibana about 3000-3500 reqest. All node is Ubuntu Os 18.04 used 1.9 version of ELK stack.
When i used 4 node(3 node 2core, one node 4 core, different HEAP size) for logstash and one server for elastic(8 code 22G HEAP) the rps 27-30K.But cpu doesn't used more then 2 core.(this test was in standart configuration ELK)
Logstash input UDP gelf plugin and send it to elastic with one index => "nginx-%{+YYYY.MM.dd}". In input it have Big json from our backend prepare from nxlog to gelf output.
I tested different configuration logstash.yml and different node, it never hight about 4K. I know that in server reqest about 8k(from monitoring in zabbix).
Question How can i debbug why logstash isn't use all resource in node.I saw logstash metrick but i can't see how mane reqest in and out because in stats is logstash document.