# Reduce filebeat cpu cost **URL:** \ **Category:** Beats\ **Created:** [January 9, 2017, 5:02am UTC](https://discuss.elastic.co/t/reduce-filebeat-cpu-cost/70914 "2017-01-09T05:02:03Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ginger](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@ginger](https://discuss.elastic.co/u/ginger)\ **Post date:** [January 9, 2017, 5:02am UTC](https://discuss.elastic.co/t/reduce-filebeat-cpu-cost/70914/1 "2017-01-09T05:02:03Z")
Hi, I use FileBeat to read a large volume of log, and then send data to ElasticSearch directly. In my case, I found filebeat consume too much CPU resource. When sending 120,000 events/s, filebeat cost 320% CPU resource. So I try to profile filebeat with net/http/pprof, and find a lot of resource is used in json serialization and common.ConvertToGenericEvent(event). Considering simplicity, I replace some json serialization with string concat and comment out common.ConvertToGenericEvent(event). After that, filebeat can send 120,000 events/s with only 150% CPU resource. As the above modification isn't a common resolution, I wonder if somebody is already working on it? ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [January 9, 2017, 1:54pm UTC](https://discuss.elastic.co/t/reduce-filebeat-cpu-cost/70914/2 "2017-01-09T13:54:41Z")
Right, the JSON serialization and `ConvertToGenericEvent` are slow because they use reflection a lot. We need, for now, the `ConvertToGenericEvent` in order to support all the community beats. We might be able to do reflection only in `ConvertToGenericEvent` though, because after that we know that the events have a short list of possible types. If you want to open a pull request with your approach, we can discuss it in more detail. In general, we postponed this optimization because we want to move away from `map[string]interface{}` as the type for events and use something that gives us more type safeness. This will make this optimization easier and safer. ---
**Author:** ![ginger](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@ginger](https://discuss.elastic.co/u/ginger)\ **Post date:** [January 10, 2017, 1:54am UTC](https://discuss.elastic.co/t/reduce-filebeat-cpu-cost/70914/3 "2017-01-10T01:54:49Z")
> [@tudor](#): > > If you want to open a pull request with your approach, we can discuss it in more detail. Yes, because I want that my users can just download an official filebeat whenever they use it. I think we need to think about an elegant resolution. Is it good to add a switch for ConvertToGenericEvent? User who doesn't need it can easily configure it to false. For JSON serialization, now I replace the following with string concat: MarshalJSON() in libbeat/common/datetime.go bulkMeta in libbeat/outputs/elasticsearch/client.go but I don't have a better idea for event serialization, because it contains many different type values. Do you have any good idea or another elegant resolution? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [January 10, 2017, 11:51am UTC](https://discuss.elastic.co/t/reduce-filebeat-cpu-cost/70914/4 "2017-01-10T11:51:59Z")
We're aware of some potential improvements in event encoding. Mostly reduce need for reflection and thusly reduce amount of garbage being generated. The logstash ouput already uses a custom json encoder, which has been shown to be somewhat faster: [https://github.com/elastic/beats/blob/master/libbeat/outputs/logstash/json.go](https://github.com/elastic/beats/blob/master/libbeat/outputs/logstash/json.go) The logstash encoder relies on the fact, all events being normalized when entering the pipeline. That is, we do know about all potential types an events value can have. The solution for the logstash output is basically a 'hack', but we might consider using it for the other outputs as well. A better solution would try to generate even less garbage (event normalization and event generation in filebeat do allocate temporary events). [Benchmarks](https://gist.github.com/urso/4a0f0c63fa9d20e71e812c3fc612b6e9#file-results) for some early experiments. The `BenchmarkXJSONEnc` uses some experimental (not yet finished) JSON encoder (only beat by optimized MessagePack encoder). But don't expect improvements/changes soonish, as there is some more groundwork to do, in order to optimize/improve the publisher pipeline in beats. The ultimate goal is to not allocate any temporary objects in filebeat if no processors are configured. ---
**Author:** ![ginger](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@ginger](https://discuss.elastic.co/u/ginger)\ **Post date:** [January 10, 2017, 12:07pm UTC](https://discuss.elastic.co/t/reduce-filebeat-cpu-cost/70914/5 "2017-01-10T12:07:54Z")
That sounds great! If some temporary solutions provided, it couldn't be better! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 30, 2017, 5:02am UTC](https://discuss.elastic.co/t/reduce-filebeat-cpu-cost/70914/6 "2017-01-30T05:02:04Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.