I have done a benchmarking for logstash v 6.3.2 and ask for help interpreting the result.
Benchmarking in logstash under couple of conditions and saw drastic decrease with Events Emitted Rate . All logs were sent from filebeat to logstash .
Server Condition
- Filebeat server
- KVM Guest
- OS: RHEL7.4
- CPU Threads: 1
- Memory: 4GB
- Logstash server
- KVM Guest
- OS: RHEL7.4
- CPU Threads: 1
- Memory: 4GB
Raw Data condition
-
Blue Circle
NASA accesslog -
Red Circle
Server Fault (Posts.xml)
Logstash pipeline condition
- Left Chart
- Input: beats
- Output: file (/dev/null)
- Filter: None
- Right Chart
- Input: beats
- Output: elasticsearch
- Filter: None
Result
-
Emitted Rate Result
-
CPU Result
-
Event Latency
-
GC duration (jstat)
Question
Events emitted rate has drastically changed when elasticsearch output was used. Since the re was an increase with Event Latency and decrease with GC duration , it looks like events were stucked somewhere between filter or output queue.
Would it be correct to understand the decrease of Emitted Rate was caused by higher latency with network service time for sending bulk requests compared to simple file output ?
I appreciate if I could get advice with clarifying my understanding.
Best Regards,
Yu Watanabe