How does elasticsearch output affect the logstash pipeline performance?

Elastic Stack Logstash
1

I have done a benchmarking for logstash v 6.3.2 and ask for help interpreting the result.

Benchmarking in logstash under couple of conditions and saw drastic decrease with Events Emitted Rate . All logs were sent from filebeat to logstash .

Server Condition

  1. Filebeat server
  • KVM Guest
  • OS: RHEL7.4
  • CPU Threads: 1
  • Memory: 4GB
  1. Logstash server
  • KVM Guest
  • OS: RHEL7.4
  • CPU Threads: 1
  • Memory: 4GB

Raw Data condition

  1. Blue Circle
    NASA accesslog

  2. Red Circle
    Server Fault (Posts.xml)

Logstash pipeline condition

  1. Left Chart
  • Input: beats
  • Output: file (/dev/null)
  • Filter: None
  1. Right Chart
  • Input: beats
  • Output: elasticsearch
  • Filter: None

Result

  1. Emitted Rate Result
    image

  2. CPU Result
    image

  3. Event Latency
    image

  4. GC duration (jstat)
    image

Question

Events emitted rate has drastically changed when elasticsearch output was used. Since the re was an increase with Event Latency and decrease with GC duration , it looks like events were stucked somewhere between filter or output queue.

Would it be correct to understand the decrease of Emitted Rate was caused by higher latency with network service time for sending bulk requests compared to simple file output ?

I appreciate if I could get advice with clarifying my understanding.

Best Regards,
Yu Watanabe

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.