Thread consumption in Filebeats

[filebeater]

Hello, I have a question on Filebeat internal:
From reading Filebeat documentation "https://www.elastic.co/guide/en/beats/filebeat/current/how-filebeat-works.html",
One harvester is started for each file. I am wondering Does each harvester has its own thread?
For example, if I have 20-30 log files to watch for concurrently, FileBeat will start 20-30 threads for this?

Thanks!
yan

[tudor (Tudor Golubenco)]

The Go runtime works a bit differently. Each harvester is a goroutine. These goroutines get scheduled on a pool of OS threads. The number of OS threads for the whole of Filebeat can be controlled from the GOMAXPROCS env variable and it defaults to the number of CPU cores on the machine.

[filebeater]

@tudor, thanks for the pointer!
would the performance of filebeat be compromised if I set GOMAXPROCS to 1? or can I set GOMAXPROCS to be 1, expecting reasonable performance? We are fine with some log processing lag as long as it is less than, say, 1 minute.

I am wondering what is the rationale of setting its default to the number of CPU cores on the machine?
Filebeat is supposed to be deployed on edge machines where the actual service is running. I would want to and minimize the Filebeat overhead and save the maximum CPU power to my service, which requires CPU and memory bandwidth.

[tudor (Tudor Golubenco)]

What we usually recommend is setting the nice value to 19 for the Filebeat process, so it will read the log files as fast as there are free resources, while still giving way to the more application services. Generally giving it 1 core should be plenty, but you never know when an application starts writing tens of thousands of log lines per second and you don't want to lag. It's perfectly fine to limit it to one core, if you prefer, or you can use cgroup limits to restrict it even more.