Performance hit when multiple filebeats are sending to same ES

[hjazz6]

@BenB196

Would you be able to provide the average IOPS on the SSDs?

I ran iostat -xd 3 /dev/sda over 25 minutes to get a more accurate reading. After averaging the numbers, I got the following.

Average w/s: 313.93
Average IOPS (r/s + w/s): 314.78
Max w/s: 1098

Average wkB/s: 68370.74
Max wkB/s: 355846.67

Average w_await: 0.691
Average %util: 3.38

@BenB196

When you see the single 28K/s Filebeat, does it have a backlog/dropped events, or is it able to send/process all data it receives?

There is always packet drop. In this case, the packet drop for single 28K/s filebeat was 33%, while the packet drop for two filebeat at 42K/s was higher at 43%.

@BenB196

Unless you are using a coordinating node, I'd make sure that you setup Filebeat Elasticsearch output to have all of your Elasticsearch nodes listed, so that Filebeat can use Round Robin output load distribution.

Thanks for this. I just realized that my Filebeat was only pointing to one ES. I'll add the other ES and see if there's any improvement.