I use FileBeat to read a large volume of log, and then send data to ElasticSearch directly.
In my case, I found filebeat consume too much CPU resource. When sending 120,000 events/s, filebeat cost 320% CPU resource.
So I try to profile filebeat with net/http/pprof, and find a lot of resource is used in json serialization and common.ConvertToGenericEvent(event). Considering simplicity, I replace some json serialization with string concat and comment out common.ConvertToGenericEvent(event). After that, filebeat can send 120,000 events/s with only 150% CPU resource.
As the above modification isn't a common resolution, I wonder if somebody is already working on it?