# Filebeat makes too many API calls choking and bringing down the Kubernetes cluster

**URL:** <https://discuss.elastic.co/t/filebeat-makes-too-many-api-calls-choking-and-bringing-down-the-kubernetes-cluster/293797>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [January 8, 2022, 8:05am UTC](https://discuss.elastic.co/t/filebeat-makes-too-many-api-calls-choking-and-bringing-down-the-kubernetes-cluster/293797 "2022-01-08T08:05:02Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ayush\_mundra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mundra/32/100014_2.png) [@ayush\_mundra](https://discuss.elastic.co/u/ayush_mundra)\
**Post date:** [January 8, 2022, 8:05am UTC](https://discuss.elastic.co/t/filebeat-makes-too-many-api-calls-choking-and-bringing-down-the-kubernetes-cluster/293797/1 "2022-01-08T08:05:03Z")

</div>

Hi,

We have filebeat version: `7.12.0` running as a daemonset on all our production K8s clusters. Recently, we faced a catastrophic situation where our whole K8s cluster with around 300+ nodes was brought down by filebeat.

We have configured filebeat with around 35 different `input.d` configurations with most of them being either `type: container` or `type: log`.

From one of those 35 input.d configs, we configured a wrong regex on `exclude_files` which started to throw errors like:

```auto
Error creating runner from config: error parsing regexp: missing argument to repetition operator: `*` accessing
 'exclude_files.0' (source:'/opt/filebeat/etc/inputs.d/abc.yml')

```

The configuration for that particular prospector was:

```auto
- type: log
  paths:
    - /var/log/containers/*_abc_*.log
  symlinks: true
  exclude_files: ["*xyz*"] // MALFORMED REGEX

  json.keys_under_root: false
  
  fields:
    topic: "abc"

  fields_under_root: true
  processors:
    - add_kubernetes_metadata:
        in_cluster: true
        default_matchers.enabled: false
        matchers:
        - logs_path:
            logs_path: /var/log/containers/

```

This bad regex configuration triggered hell lot of API calls to `kube-apiserver` for kubernetes\_metadata fetch, that it brought down the master node itself which eventually brought down the working of the kubernetes cluster itself.

How can we make filebeat better to handle these kind of misconfigs properly and consequently doesn't bring down the entire k8s?

What are the suggestions in general on the usage of `add_kubernetes_metadata`? Coz I think if I compile all the different input.d configurations, into a single one with just one `add_kubernetes_metadata` config, the number of requests to K8s API will significantly reduce and may help in not bringing down the entire K8s cluster. (Please let me know if that will help.)

Attaching some images to provide a gist on the increase in the number of K8s API requests.

 ![Screenshot 2022-01-08 at 1.22.14 PM](https://us1.discourse-cdn.com/elastic/original/3X/2/b/2bff3d00774b7da75f0b4af99b8299d343fc467e.png)

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [January 10, 2022, 9:36am UTC](https://discuss.elastic.co/t/filebeat-makes-too-many-api-calls-choking-and-bringing-down-the-kubernetes-cluster/293797/3 "2022-01-10T09:36:37Z")

</div>

Hi @ayush_mundra !  
What happened here is interesting but hard to spot the root cause only from the description. In general if you define the processor for each of the inputs then you will have a single processor instantiated per input with its one cache. This indeed can lead in more load compared to having one processor on Filebeat's level. Hope it helps a little bit :)!

C.

---

<div class="post-metadata">

**Author:** ![ayush\_mundra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mundra/32/100014_2.png) [@ayush\_mundra](https://discuss.elastic.co/u/ayush_mundra)\
**Post date:** [January 11, 2022, 6:21am UTC](https://discuss.elastic.co/t/filebeat-makes-too-many-api-calls-choking-and-bringing-down-the-kubernetes-cluster/293797/4 "2022-01-11T06:21:30Z")

</div>

Hi @ChrsMark  
Thanks for the reply!

> What happened here is interesting but hard to spot the root cause only from the description.

Please let me know how I can make it more elaborate and what kind of informations, you require from my end.

> In general if you define the processor for each of the inputs then you will have a single processor instantiated per input with its one cache.

Yes, that's correct and we actually tried it on our side. The number of requests got reduced by 10x and it seems that is something we will have to target with a lot of conditionals and processors.

But the biggest concern is malformed `exclude_files: ["*xyz*"] // MALFORMED REGEX`. I think this is open to bad configs and can easily be missed as was the case with us. But if it potentially brings down the entire K8s, then it is pretty scary and needs to be corrected.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 8, 2022, 8:22am UTC](https://discuss.elastic.co/t/filebeat-makes-too-many-api-calls-choking-and-bringing-down-the-kubernetes-cluster/293797/5 "2022-02-08T08:22:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.