Make Filebeat Faster

[Naruhodou (Reggie Yang)]

Hi, is there a way to make Filebeat faster? Right now when there are many logs generating at the same time, it takes several seconds before Filebeat pushes to Logstash. Are there any section in the documentation or some configs that can make Filebeat trade more resources for the speed?

Thanks!

[Christian_Dahlqvist]

How are you determining this delay? How have you established that Filebeat is the bottleneck? Where is Logstash in turn sending the data?

[Naruhodou (Reggie Yang)]

Hi Christian,

Thanks for help! I'm not sure whether my way to verify where the delay is is right or not.

Right now Filebeat are running on 10 machines. Logstash is on another machine in the same network. Let's take one line of log as an example.

Here the field '@datetime'(11:51:41,387) is when the log is generated. It is parsed from the log. The field '@timestamp'(11:51:44.801) is generated by Filebeat(I think it is?), so I think it's the time when Filebeat sent the message. And the field '@collected_time'(11:51:45,905) is a field I put at the end of Logstash. The configuration is

   ruby {
        code => 'require "date"
                 current_time = DateTime.now
                 t = current_time.strftime("%d/%m/%Y %H:%M:%S,%L")
                 event.set("@collected_time", t)'
    }

In this case, I think the gap between @timestamp and @datetime is relatively large.

[Christian_Dahlqvist]

That seems to be a delay of just a few seconds. This may be because Filebeat batches up events for improved efficiency. How many events are generated per beat per second?

[Naruhodou (Reggie Yang)]

In the past 5 mins, the average rate is less than 1.(180 logs on 5 machines in 5 mins). But as I see, the speed of generating log is not the same all the time. It's like about 4 logs at the same time, then nothing for several seconds, then another 4 logs at the same time. So the peak rate is about 4 as I see.

[Christian_Dahlqvist]

I suspect that delay is because you have a very low event rate and Filebeat is buffering. You could lower the bulk_max_size, but that could limit throughput if volumes were to pick up, as indexing very small batches is inefficient.

[Naruhodou (Reggie Yang)]

Which means Filebeat is buffering the events until it reach a certain size to achieve a better performance? Understood. Thanks!