Harvester_buffer_size - why increase it?

[mikewillis]

The documentation says what the harvest_buffer_size setting of the log input is for, but I don't see anything about why one might want to change it.

Log input | Filebeat Reference [7.17] | Elastic

I noticed that a Filebeat 7.17 instance on a CentOS 7 server that is sending a lot of data from multiple files to the Logstash layer our of logging system was using 10s of GB of memory and causing a problem with memory exhaustion. harvester_buffer_size had been set to 1048576000 but I don't know why. Commenting out that bit of the config has greatly reduced the memory usage and casual observation shows data still reaching our cluster at the same rate. Other than because you would want Filebeat to use more memory, why would one increase harvester_buffer_size above the default?

[Andrea_Spacca]

hi @mikewillis

tweaking harvester_buffer_size value has impact on the performance. When optimisation is a concern it is possible to set the most efficient value that can vary according to the environment.

[mikewillis]

OK, so I guess increasing the value is supposed to increase the performance. Is that right? How would one know what the most efficient value might be?