# Filebeat CPU shoots 100 times when sending to kafka topic using a field **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [August 31, 2018, 4:58pm UTC](https://discuss.elastic.co/t/filebeat-cpu-shoots-100-times-when-sending-to-kafka-topic-using-a-field/146898 "2018-08-31T16:58:55Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![krunalvora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krunalvora/32/35031_2.png) [@krunalvora](https://discuss.elastic.co/u/krunalvora)\ **Post date:** [August 31, 2018, 4:58pm UTC](https://discuss.elastic.co/t/filebeat-cpu-shoots-100-times-when-sending-to-kafka-topic-using-a-field/146898/1 "2018-08-31T16:58:55Z")
When using the filebeat config output like: ```auto output.kafka: hosts: - :9092 topic: 'topic' ``` The CPU usage of filebeat is around ~15m in kubernetes. But when I try to dynamically set the topic using the fields of the event: ```auto output.kafka: hosts: - :9092 topic: 'topic-%{[kubernetes.namespace]}' ``` the CPU shoots to \>2000mi and the logs stop flowing to kafka within a minute. Could anyone help understand this behavior and whether I can get the CPU down to feasible limits? Filebeat version: 6.2.4 Kafka version: 0.1.1 ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [September 3, 2018, 7:49am UTC](https://discuss.elastic.co/t/filebeat-cpu-shoots-100-times-when-sending-to-kafka-topic-using-a-field/146898/2 "2018-09-03T07:49:11Z")
In the first case `topic` is a constant selector. It means that Filebeat sets the topic of events to a constant string always. There is no need to evaluate/run anything to determine the topic of the event. In the second case a dynamic `topic` is set. This requires Filebeat to evaluate a format string and find a matching field for each event. The value of the matching field is going to be the topic. This process requires more resources than the first one. However, log flow shouldn't be stopped in this case. Could you please share your full config formatted using `` and if possible debug logs in case of dynamic topics? ---
**Author:** ![krunalvora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krunalvora/32/35031_2.png) [@krunalvora](https://discuss.elastic.co/u/krunalvora)\ **Post date:** [September 3, 2018, 10:41pm UTC](https://discuss.elastic.co/t/filebeat-cpu-shoots-100-times-when-sending-to-kafka-topic-using-a-field/146898/3 "2018-09-03T22:41:49Z")
Thanks Neomi. I am running filebeat in a kubernetes cluster providing limited resources for CPU(100m) and Memory(100Mi) and the CPU usage shoots upto 2000m within minutes of deployment of filebeat daemonset. Here is the config: ``` filebeat.yml: |- filebeat.config: prospectors: # Mounted `filebeat-prospectors` configmap: path: ${path.config}/prospectors.d/*.yml # Reload prospectors configs as they change: reload.enabled: false modules: path: ${path.config}/modules.d/*.yml # Reload module configs as they change: reload.enabled: false output.kafka: hosts: - :9092 topic: 'test-%{[kubernetes.namespace]}' ``` And here is the prospector file: ``` kubernetes.yml: |- - type: log paths: - /var/lib/docker/containers/*/*.log json.message_key: log json.keys_under_root: true processors: - add_kubernetes_metadata: in_cluster: true namespace: ${POD_NAMESPACE} ``` I can understand the fact that the dynamic topic evaluation would take up more resources. But I want to confirm if that increase would be 100 times. Please let me know if any options setting in the config could cause this problem. ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [September 4, 2018, 10:31am UTC](https://discuss.elastic.co/t/filebeat-cpu-shoots-100-times-when-sending-to-kafka-topic-using-a-field/146898/4 "2018-09-04T10:31:02Z")
Hi @krunalvora, In order to address this, we should discard possible sources for this issue one by one. Can you confirm that removing the dynamic topic setting the CPU stays constant? Does it happen in all cases? Best regards ---
**Author:** ![krunalvora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krunalvora/32/35031_2.png) [@krunalvora](https://discuss.elastic.co/u/krunalvora)\ **Post date:** [September 4, 2018, 6:14pm UTC](https://discuss.elastic.co/t/filebeat-cpu-shoots-100-times-when-sending-to-kafka-topic-using-a-field/146898/5 "2018-09-04T18:14:18Z")
@exekias Yes, removing the dynamic topic config brings down the CPU to around 30m on a constant basis. ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [September 5, 2018, 11:03pm UTC](https://discuss.elastic.co/t/filebeat-cpu-shoots-100-times-when-sending-to-kafka-topic-using-a-field/146898/6 "2018-09-05T23:03:29Z")
I would say we can consider this a bug. Could you please open a new report here? [https://github.com/elastic/beats/issues/new](https://github.com/elastic/beats/issues/new) Please provide as much details as possible, also describe how to reproduce the issue. All this should be helpful to debug the issue. Br, ---
**Author:** ![krunalvora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krunalvora/32/35031_2.png) [@krunalvora](https://discuss.elastic.co/u/krunalvora)\ **Post date:** [September 24, 2018, 8:18pm UTC](https://discuss.elastic.co/t/filebeat-cpu-shoots-100-times-when-sending-to-kafka-topic-using-a-field/146898/7 "2018-09-24T20:18:37Z")
@exekias Thanks for the suggestion! Creating a new issue now. ---
**Author:** ![krunalvora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krunalvora/32/35031_2.png) [@krunalvora](https://discuss.elastic.co/u/krunalvora)\ **Post date:** [September 24, 2018, 8:29pm UTC](https://discuss.elastic.co/t/filebeat-cpu-shoots-100-times-when-sending-to-kafka-topic-using-a-field/146898/8 "2018-09-24T20:29:10Z")
@exekias > ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 22, 2018, 8:29pm UTC](https://discuss.elastic.co/t/filebeat-cpu-shoots-100-times-when-sending-to-kafka-topic-using-a-field/146898/9 "2018-10-22T20:29:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.