# Registry blows up after filebeat container restarts

**URL:** https://discuss.elastic.co/t/registry-blows-up-after-filebeat-container-restarts/293572
**Category:** Beats
**Tags:** docker, filebeat
**Created:** [January 5, 2022, 5:51pm UTC](https://discuss.elastic.co/t/registry-blows-up-after-filebeat-container-restarts/293572 "2022-01-05T17:51:58Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![ivankovnikita1992](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivankovnikita1992/32/99885_2.png) [@ivankovnikita1992](https://discuss.elastic.co/u/ivankovnikita1992)
#### Post date: [January 5, 2022, 5:51pm UTC](https://discuss.elastic.co/t/registry-blows-up-after-filebeat-container-restarts/293572/1 "2022-01-05T17:51:58Z")

</div>

Hello there,

I've already read a lot of docs and issues but haven't found any solution for my problem, so hopefully someone can help...

Our implementation is pretty simple - filebeat is set as daemonset for kubernetes cluster which means that each node has the filebeat pod which gets logs of the containers with specific label and forwards them to elastic

The configuration of filebeat is as the following

```auto
 setup.ilm:
        enabled: false
      setup.template:
        enabled: false
        name: "%{[kubernetes][labels][elastic-index]}"
        pattern: "%{[kubernetes][labels][elastic-index]}-*"
      processors:
      - decode_json_fields:
          fields: ["message"]
          process_array: true
          max_depth: 10
          target: ""
          overwrite_keys: false
          add_error_key: false
      - add_cloud_metadata: ~

      filebeat.inputs:
      - type: container
        paths:
        - '/var/lib/docker/containers/*/*.log'
        processors:
        - add_kubernetes_metadata:
            in_cluster: true
        - drop_event:
            when:
              not:
                regexp:
                  kubernetes.labels.elastic-index: ".*"

```

The problem we have is that when filebeat pod get restarted(OOM for example), it goes into some "crazy mode" - it updates registry file every second

```auto
4167578795.json active.dat log.json meta.json
sh-4.2# ls
4167601926.json active.dat log.json meta.json
sh-4.2# ls
4167625055.json active.dat log.json meta.json
sh-4.2# ls
4167648187.json active.dat log.json meta.json
sh-4.2# ls
4167648187.json active.dat log.json meta.json

```

and that file contains thousands of lines

```auto
sh-4.2# cat 4170146226.json | wc
   2705 2706 1147578

```

but only for few log files

```auto
cat 4170146226.json | jq -r '.[].source' | sort -u | wc
      60 60 9960

```

This is the output from the node of the same k8s cluster but in "right" mode

```auto
4675893.json active.dat log.json meta.json
sh-4.2# ls
4675893.json active.dat log.json meta.json
sh-4.2# ls
4675893.json active.dat log.json meta.json
sh-4.2# ls
4675893.json active.dat log.json meta.json
sh-4.2# ls
4675893.json active.dat log.json meta.json
sh-4.2# cat 4675893.json | wc
     35 36 14383

```

And for sure the logs from the node with this "crazy" filebeat pod are lost.

We're using

```auto
image: docker.elastic.co/beats/filebeat
  imageTag: 7.10.1

```

Is there any idea how to debug it? (besides increasing the memory limits for the pod to prevent it from getting killed)

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 2, 2022, 7:52pm UTC](https://discuss.elastic.co/t/registry-blows-up-after-filebeat-container-restarts/293572/2 "2022-02-02T19:52:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
