Filebeat CPU shoots 100 times when sending to kafka topic using a field

Elastic Stack Beats
1

When using the filebeat config output like:

    output.kafka:
      hosts:
      - <kafka>:9092
      topic: 'topic'

The CPU usage of filebeat is around ~15m in kubernetes.

But when I try to dynamically set the topic using the fields of the event:

    output.kafka:
      hosts:
      - <kafka>:9092
      topic: 'topic-%{[kubernetes.namespace]}'

the CPU shoots to >2000mi and the logs stop flowing to kafka within a minute.

Could anyone help understand this behavior and whether I can get the CPU down to feasible limits?

Filebeat version: 6.2.4
Kafka version: 0.1.1

2

In the first case topic is a constant selector. It means that Filebeat sets the topic of events to a constant string always. There is no need to evaluate/run anything to determine the topic of the event.

In the second case a dynamic topic is set. This requires Filebeat to evaluate a format string and find a matching field for each event. The value of the matching field is going to be the topic. This process requires more resources than the first one.

However, log flow shouldn't be stopped in this case. Could you please share your full config formatted using </> and if possible debug logs in case of dynamic topics?

3

Thanks Neomi.

I am running filebeat in a kubernetes cluster providing limited resources for CPU(100m) and Memory(100Mi) and the CPU usage shoots upto 2000m within minutes of deployment of filebeat daemonset.

Here is the config:

filebeat.yml: |-
    filebeat.config:
      prospectors:
        # Mounted `filebeat-prospectors` configmap:
        path: ${path.config}/prospectors.d/*.yml
        # Reload prospectors configs as they change:
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false
    output.kafka:
      hosts:
        - <kafka-endpoint>:9092
      topic: 'test-%{[kubernetes.namespace]}'

And here is the prospector file:

kubernetes.yml: |-
    - type: log
      paths:
        - /var/lib/docker/containers/*/*.log
      json.message_key: log
      json.keys_under_root: true
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            namespace: ${POD_NAMESPACE}

I can understand the fact that the dynamic topic evaluation would take up more resources. But I want to confirm if that increase would be 100 times.
Please let me know if any options setting in the config could cause this problem.

4

Hi @krunalvora,

In order to address this, we should discard possible sources for this issue one by one. Can you confirm that removing the dynamic topic setting the CPU stays constant? Does it happen in all cases?

Best regards

5

@exekias
Yes, removing the dynamic topic config brings down the CPU to around 30m on a constant basis.

6

I would say we can consider this a bug. Could you please open a new report here? https://github.com/elastic/beats/issues/new

Please provide as much details as possible, also describe how to reproduce the issue. All this should be helpful to debug the issue.

Br,

7

@exekias Thanks for the suggestion!

Creating a new issue now.

8

@exekias

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.