# Filebeat memory keeps increasing until OOM

**URL:** <https://discuss.elastic.co/t/filebeat-memory-keeps-increasing-until-oom/365880>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 2, 2024, 5:42am UTC](https://discuss.elastic.co/t/filebeat-memory-keeps-increasing-until-oom/365880 "2024-09-02T05:42:34Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![syunka.tyo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syunka.tyo/32/137211_2.png) [@syunka.tyo](https://discuss.elastic.co/u/syunka.tyo)\
**Post date:** [September 2, 2024, 5:42am UTC](https://discuss.elastic.co/t/filebeat-memory-keeps-increasing-until-oom/365880/1 "2024-09-02T05:42:34Z")

</div>

Hello all

We are facing persistent Out of Memory(OOM) issue with filebeat across our kubernetes environment. We initially encountered the issue with Filebeat version 8.5.1. Follwing the suggestions from a topic([Filebeat: "memory leak" via filebeat.autodiscover and \>\>200.000 goroutines](https://discuss.elastic.co/t/filebeat-memory-leak-via-filebeat-autodiscover-and-200-000-goroutines/322082)) on your forum regarding a memory leak via Filebeat autodiscover, we tested versions 8.9.0 and 8.15.0 as well. Unfortunately, the issue persistes across all these version.

Could you please help to take a look?  
Backgound:

1. Filebeat version: 8.5.1/8.9.0/8.15.0

2. Filebeat config file:

3. 200 cronjob simulation in parallel  
In order to simulate the issue, we manually create 200 cronjob in parallel to the namespace where filebeat is gathering log.

4. Please find the below graph  
We have deployed filebeat on three Kubernete nodes and each instance consistently encounters OOM problems. Each instance has the similar graph as below:  

5. Gather the heap pprof for your help  
In order to determine the root cause ASAP. We gathered the heap by pprof and pickup two bin file. One is picked up at the start of the test while second one is picked up just before OOM.  
11：25

```auto
File: filebeat
Build ID: 0685fb39bd17876b71744aebb3b7fd7fd0dd242f
Type: inuse_space
Time: Sep 2, 2024 at 11:29am (JST)
Showing nodes accounting for 99.93MB, 89.20% of 112.03MB total
Dropped 93 nodes (cum <= 0.56MB)
      flat flat% sum% cum cum%
   26.51MB 23.66% 23.66% 26.51MB 23.66% github.com/elastic/go-structform/gotype.(*unfolderMapIfc).put (inline)
   16.01MB 14.29% 37.95% 16.01MB 14.29% github.com/elastic/elastic-agent-libs/mapstr.M.Clone
   11.51MB 10.27% 48.22% 11.51MB 10.27% reflect.unsafe_NewArray
    7.78MB 6.95% 55.17% 7.78MB 6.95% github.com/elastic/beats/v7/filebeat/input/file.(*States).GetStates
       6MB 5.36% 60.53% 6MB 5.36% github.com/goccy/go-json/internal/decoder.init.0
       4MB 3.57% 64.10% 4MB 3.57% github.com/elastic/beats/v7/filebeat/registrar.readStatesFrom.func1
       4MB 3.57% 67.67% 36.01MB 32.14% github.com/elastic/beats/v7/filebeat/registrar.writeStates
    3.01MB 2.68% 70.36% 3.01MB 2.68% github.com/goccy/go-json/internal/encoder.init.0
       3MB 2.68% 73.03% 3MB 2.68% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).literalStore
    2.50MB 2.23% 75.27% 2.50MB 2.23% reflect.New
    2.50MB 2.23% 77.50% 2.50MB 2.23% encoding/json.(*decodeState).literalStore
       2MB 1.79% 79.28% 2MB 1.79% k8s.io/apimachinery/pkg/apis/meta/v1.(*FieldsV1).UnmarshalJSON
       2MB 1.79% 81.07% 2MB 1.79% github.com/elastic/go-structform/gotype.makeFieldFold (inline)
    1.50MB 1.34% 82.41% 18.01MB 16.07% github.com/elastic/go-structform/gotype.(*unfolderMapIfc).OnString
    1.05MB 0.94% 83.35% 1.05MB 0.94% github.com/elastic/beats/v7/filebeat/input/file.(*States).SetStates
    1.05MB 0.94% 84.29% 1.05MB 0.94% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.loadDataFile.func1
       1MB 0.89% 85.18% 1MB 0.89% github.com/elastic/elastic-agent-libs/mapstr.M.Put
       1MB 0.89% 86.07% 12.01MB 10.72% github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).publish
       1MB 0.89% 86.97% 1MB 0.89% strings.(*Builder).grow
    0.50MB 0.45% 87.41% 1.50MB 1.34% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).containerPodEvents
    0.50MB 0.45% 87.86% 3MB 2.68% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*Resource).GenerateK8s
    0.50MB 0.45% 88.31% 1MB 0.89% github.com/elastic/go-structform/gotype.NewIterator
    0.50MB 0.45% 88.75% 3MB 2.68% encoding/json.(*decodeState).object
    0.50MB 0.45% 89.20% 2.50MB 2.23% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.GenerateMap
         0 0% 89.20% 3MB 2.68% encoding/json.(*Decoder).Decode
         0 0% 89.20% 3MB 2.68% encoding/json.(*decodeState).array
         0 0% 89.20% 3MB 2.68% encoding/json.(*decodeState).unmarshal
         0 0% 89.20% 3MB 2.68% encoding/json.(*decodeState).value
         0 0% 89.20% 9.11MB 8.13% github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run
         0 0% 89.20% 12.01MB 10.72% github.com/elastic/beats/v7/filebeat/beater.(*countingClient).Publish
         0 0% 89.20% 4.05MB 3.62% github.com/elastic/beats/v7/filebeat/beater.(*filebeatStore).Access
         0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/filebeat/channel.(*onCreateFactory).CheckConfig
         0 0% 89.20% 12.01MB 10.72% github.com/elastic/beats/v7/filebeat/channel.(*outlet).OnEvent
         0 0% 89.20% 12.01MB 10.72% github.com/elastic/beats/v7/filebeat/channel.SubOutlet.func1
         0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/filebeat/input.(*RunnerFactory).CheckConfig
         0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/filebeat/input.(*RunnerFactory).Create
         0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/filebeat/input/v2/compat.composeFactory.CheckConfig
         0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).GetStates (inline)
         0 0% 89.20% 39.90MB 35.62% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Run
         0 0% 89.20% 5.05MB 4.51% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start
         0 0% 89.20% 39.90MB 35.62% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start.func1
         0 0% 89.20% 39.90MB 35.62% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).commitStateUpdates
         0 0% 89.20% 5.05MB 4.51% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).loadStates
         0 0% 89.20% 4.05MB 3.62% github.com/elastic/beats/v7/filebeat/registrar.New
         0 0% 89.20% 4MB 3.57% github.com/elastic/beats/v7/filebeat/registrar.readStatesFrom
         0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).handleStart
         0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).worker
         0 0% 89.20% 5MB 4.46% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*Provider).publish
         0 0% 89.20% 1MB 0.89% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnAdd
         0 0% 89.20% 1.50MB 1.34% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnDelete
         0 0% 89.20% 6.50MB 5.80% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnUpdate
         0 0% 89.20% 9MB 8.04% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).emit
         0 0% 89.20% 2.50MB 2.23% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).podEvent
         0 0% 89.20% 5MB 4.46% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).publishAll
         0 0% 89.20% 6.50MB 5.80% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).unlockedUpdate
         0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/libbeat/cfgfile.multiplexedFactory.CheckConfig
         0 0% 89.20% 9.11MB 8.13% github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1
         0 0% 89.20% 9.11MB 8.13% github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch
         0 0% 89.20% 9.11MB 8.13% github.com/elastic/beats/v7/libbeat/cmd/instance.Run
         0 0% 89.20% 9.11MB 8.13% github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1
         0 0% 89.20% 28.01MB 25.00% github.com/elastic/beats/v7/libbeat/common/transform/typeconv.(*Converter).Convert
         0 0% 89.20% 28.01MB 25.00% github.com/elastic/beats/v7/libbeat/common/transform/typeconv.Convert
         0 0% 89.20% 12.01MB 10.72% github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).Publish
         0 0% 89.20% 10.51MB 9.38% github.com/elastic/beats/v7/libbeat/publisher/processing.(*builder).Create.makeAddDynMetaProcessor.func4
         0 0% 89.20% 10.51MB 9.38% github.com/elastic/beats/v7/libbeat/publisher/processing.(*builder).Create.makeAddDynMetaProcessor.newAnnotateProcessor.func6
         0 0% 89.20% 11.01MB 9.82% github.com/elastic/beats/v7/libbeat/publisher/processing.(*group).Run
         0 0% 89.20% 10.51MB 9.38% github.com/elastic/beats/v7/libbeat/publisher/processing.(*processorFn).Run
         0 0% 89.20% 4.05MB 3.62% github.com/elastic/beats/v7/libbeat/statestore.(*Registry).Get
         0 0% 89.20% 4MB 3.57% github.com/elastic/beats/v7/libbeat/statestore.(*Store).Each
         0 0% 89.20% 32.01MB 28.57% github.com/elastic/beats/v7/libbeat/statestore.(*Store).Set
         0 0% 89.20% 4.05MB 3.62% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*Registry).Access
         0 0% 89.20% 4MB 3.57% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*diskstore).LogOperation
         0 0% 89.20% 3MB 2.68% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*jsonEncoder).Encode (inline)
         0 0% 89.20% 1MB 0.89% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*jsonEncoder).reset
         0 0% 89.20% 4MB 3.57% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).Each
         0 0% 89.20% 32.01MB 28.57% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).Set
         0 0% 89.20% 4MB 3.57% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).logOperation
         0 0% 89.20% 4.05MB 3.62% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.loadDataFile
         0 0% 89.20% 1MB 0.89% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.newJSONEncoder (inline)
         0 0% 89.20% 4.05MB 3.62% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.openStore
         0 0% 89.20% 4.05MB 3.62% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.readDataFile
         0 0% 89.20% 9MB 8.04% github.com/elastic/elastic-agent-autodiscover/kubernetes.(*watcher).Start.func1
         0 0% 89.20% 9MB 8.04% github.com/elastic/elastic-agent-autodiscover/kubernetes.(*watcher).process
         0 0% 89.20% 2MB 1.79% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*node).GenerateFromName
         0 0% 89.20% 2MB 1.79% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*node).GenerateK8s
         0 0% 89.20% 3MB 2.68% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*pod).Generate
         0 0% 89.20% 3MB 2.68% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*pod).GenerateK8s
         0 0% 89.20% 1MB 0.89% github.com/elastic/elastic-agent-autodiscover/utils.DeDot (inline)
         0 0% 89.20% 31.01MB 27.68% github.com/elastic/go-structform/gotype.(*Iterator).Fold (inline)
         0 0% 89.20% 18.01MB 16.07% github.com/elastic/go-structform/gotype.(*unfoldCtx).OnString
         0 0% 89.20% 10MB 8.93% github.com/elastic/go-structform/gotype.(*unfoldCtx).OnUint64
         0 0% 89.20% 10MB 8.93% github.com/elastic/go-structform/gotype.(*unfolderMapIfc).OnUint64
         0 0% 89.20% 2.50MB 2.23% github.com/elastic/go-structform/gotype.buildFieldFold
         0 0% 89.20% 28.01MB 25.00% github.com/elastic/go-structform/gotype.buildFieldFold.makeFieldFold.func1
         0 0% 89.20% 31.01MB 27.68% github.com/elastic/go-structform/gotype.foldAnyReflect
         0 0% 89.20% 31.01MB 27.68% github.com/elastic/go-structform/gotype.foldInterfaceValue
         0 0% 89.20% 3MB 2.68% github.com/elastic/go-structform/gotype.getReflectFold
         0 0% 89.20% 3MB 2.68% github.com/elastic/go-structform/gotype.getReflectFoldStruct
         0 0% 89.20% 28.01MB 25.00% github.com/elastic/go-structform/gotype.getReflectFoldStruct.makeStructFold.func2
         0 0% 89.20% 28.01MB 25.00% github.com/elastic/go-structform/gotype.getReflectFoldStruct.makeStructFold.makeFieldsFold.func3
         0 0% 89.20% 2.50MB 2.23% github.com/elastic/go-structform/gotype.getStructFieldsFolds
         0 0% 89.20% 18.01MB 16.07% github.com/elastic/go-structform/gotype.reFoldString
         0 0% 89.20% 10MB 8.93% github.com/elastic/go-structform/gotype.reFoldUint64
         0 0% 89.20% 9.11MB 8.13% github.com/spf13/cobra.(*Command).Execute
         0 0% 89.20% 9.11MB 8.13% github.com/spf13/cobra.(*Command).ExecuteC
         0 0% 89.20% 9.11MB 8.13% github.com/spf13/cobra.(*Command).execute
         0 0% 89.20% 1.01MB 0.9% google.golang.org/protobuf/internal/filedesc.Builder.Build
         0 0% 89.20% 1.01MB 0.9% google.golang.org/protobuf/internal/filedesc.newRawFile
         0 0% 89.20% 1.53MB 1.36% google.golang.org/protobuf/internal/filetype.Builder.Build
         0 0% 89.20% 2MB 1.79% k8s.io/apimachinery/pkg/runtime.(*Scheme).New
         0 0% 89.20% 18.51MB 16.52% k8s.io/apimachinery/pkg/runtime.Decode (inline)
         0 0% 89.20% 2MB 1.79% k8s.io/apimachinery/pkg/runtime.UseOrCreateObject
         0 0% 89.20% 19.51MB 17.42% k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode
         0 0% 89.20% 20.02MB 17.87% k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode
         0 0% 89.20% 18.01MB 16.08% k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).unmarshal
         0 0% 89.20% 9.50MB 8.48% k8s.io/apimachinery/pkg/util/wait.BackoffUntil
         0 0% 89.20% 9.50MB 8.48% k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
         0 0% 89.20% 9.50MB 8.48% k8s.io/apimachinery/pkg/util/wait.JitterUntil
         0 0% 89.20% 9.50MB 8.48% k8s.io/apimachinery/pkg/util/wait.Until
         0 0% 89.20% 18.51MB 16.52% k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive
         0 0% 89.20% 1MB 0.89% k8s.io/client-go/kubernetes/typed/core/v1.(*pods).List
         0 0% 89.20% 1MB 0.89% k8s.io/client-go/rest.Result.Into
         0 0% 89.20% 18.51MB 16.52% k8s.io/client-go/rest/watch.(*Decoder).Decode
         0 0% 89.20% 9.61MB 8.58% main.main
         0 0% 89.20% 11.51MB 10.27% reflect.MakeSlice
         0 0% 89.20% 15.60MB 13.92% runtime.doInit (inline)
         0 0% 89.20% 15.60MB 13.92% runtime.doInit1
         0 0% 89.20% 25.20MB 22.50% runtime.main
         0 0% 89.20% 18.01MB 16.08% sigs.k8s.io/json.UnmarshalCaseSensitivePreserveInts (inline)
         0 0% 89.20% 16.51MB 14.74% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).array
         0 0% 89.20% 18.01MB 16.08% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).object
         0 0% 89.20% 18.01MB 16.08% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).unmarshal
         0 0% 89.20% 18.01MB 16.08% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).value
         0 0% 89.20% 18.01MB 16.08% sigs.k8s.io/json/internal/golang/encoding/json.Unmarshal
         0 0% 89.20% 1MB 0.89% strings.(*Builder).Grow
         0 0% 89.20% 1MB 0.89% strings.Replace

```

14：04

```auto
File: filebeat
Build ID: 0685fb39bd17876b71744aebb3b7fd7fd0dd242f
Type: inuse_space
Time: Sep 2, 2024 at 2:05pm (JST)
Showing nodes accounting for 123.88MB, 90.74% of 136.51MB total
Dropped 103 nodes (cum <= 0.68MB)
      flat flat% sum% cum cum%
   32.51MB 23.82% 23.82% 32.51MB 23.82% github.com/elastic/go-structform/gotype.(*unfolderMapIfc).put (inline)
   19.51MB 14.29% 38.11% 19.51MB 14.29% github.com/elastic/elastic-agent-libs/mapstr.M.Clone
   11.51MB 8.43% 46.54% 11.51MB 8.43% reflect.unsafe_NewArray
    8.19MB 6.00% 52.54% 8.19MB 6.00% github.com/elastic/beats/v7/filebeat/input/file.(*States).GetStates
       6MB 4.40% 56.93% 6MB 4.40% github.com/goccy/go-json/internal/decoder.init.0
    5.03MB 3.69% 60.62% 5.03MB 3.69% github.com/elastic/beats/v7/filebeat/input/file.(*States).UpdateWithTs
    4.50MB 3.30% 63.92% 4.50MB 3.30% reflect.New
    4.50MB 3.30% 67.21% 49.01MB 35.90% github.com/elastic/beats/v7/filebeat/registrar.writeStates
    3.50MB 2.56% 69.78% 4.50MB 3.30% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).literalStore
    3.50MB 2.56% 72.34% 25.01MB 18.32% github.com/elastic/go-structform/gotype.(*unfolderMapIfc).OnString
    3.01MB 2.20% 74.55% 3.01MB 2.20% github.com/goccy/go-json/internal/encoder.init.0
    2.50MB 1.83% 76.38% 2.50MB 1.83% encoding/json.(*decodeState).literalStore
       2MB 1.47% 77.84% 2MB 1.47% github.com/elastic/elastic-agent-libs/mapstr.M.deepUpdateMap
       2MB 1.47% 79.31% 5MB 3.66% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*Resource).GenerateK8s
       2MB 1.47% 80.77% 2MB 1.47% github.com/elastic/go-structform/json.NewVisitor (inline)
       2MB 1.47% 82.24% 2MB 1.47% github.com/elastic/go-structform/gotype.makeFieldFold (inline)
       2MB 1.47% 83.70% 2MB 1.47% strings.(*Builder).grow
    1.50MB 1.10% 84.80% 1.50MB 1.10% github.com/elastic/go-structform/gotype.(*typeFoldRegistry).set (inline)
    1.50MB 1.10% 85.90% 3MB 2.20% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.GenerateMap
    1.05MB 0.77% 86.67% 1.05MB 0.77% github.com/elastic/beats/v7/filebeat/input/file.(*States).SetStates
    1.05MB 0.77% 87.44% 1.05MB 0.77% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.loadDataFile.func1
       1MB 0.73% 88.18% 1MB 0.73% k8s.io/apimachinery/pkg/apis/meta/v1.(*FieldsV1).UnmarshalJSON
       1MB 0.73% 88.91% 1MB 0.73% github.com/elastic/beats/v7/filebeat/input/log.(*Harvester).onMessage
    0.50MB 0.37% 89.28% 1MB 0.73% github.com/elastic/elastic-agent-libs/safemapstr.Put (inline)
    0.50MB 0.37% 89.64% 12.51MB 9.16% github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).publish
    0.50MB 0.37% 90.01% 3MB 2.20% encoding/json.(*decodeState).object
    0.50MB 0.37% 90.37% 11.50MB 8.43% github.com/elastic/go-structform/gotype.(*unfolderMapIfc).OnUint64
    0.50MB 0.37% 90.74% 5MB 3.66% github.com/elastic/go-structform/gotype.getStructFieldsFolds
         0 0% 90.74% 3MB 2.20% encoding/json.(*Decoder).Decode
         0 0% 90.74% 3MB 2.20% encoding/json.(*decodeState).array
         0 0% 90.74% 3MB 2.20% encoding/json.(*decodeState).unmarshal
         0 0% 90.74% 3MB 2.20% encoding/json.(*decodeState).value
         0 0% 90.74% 5.11MB 3.74% github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run
         0 0% 90.74% 4.59MB 3.37% github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run.RunnerFactoryWithCommonInputSettings.func10
         0 0% 90.74% 12.51MB 9.16% github.com/elastic/beats/v7/filebeat/beater.(*countingClient).Publish
         0 0% 90.74% 4.05MB 2.97% github.com/elastic/beats/v7/filebeat/beater.(*filebeatStore).Access
         0 0% 90.74% 4.59MB 3.37% github.com/elastic/beats/v7/filebeat/channel.(*onCreateFactory).Create
         0 0% 90.74% 12.51MB 9.16% github.com/elastic/beats/v7/filebeat/channel.(*outlet).OnEvent
         0 0% 90.74% 12.51MB 9.16% github.com/elastic/beats/v7/filebeat/channel.SubOutlet.func1
         0 0% 90.74% 1MB 0.73% github.com/elastic/beats/v7/filebeat/harvester.(*Registry).Start.func1
         0 0% 90.74% 4.59MB 3.37% github.com/elastic/beats/v7/filebeat/input.(*RunnerFactory).Create
         0 0% 90.74% 1MB 0.73% github.com/elastic/beats/v7/filebeat/input/log.(*Harvester).Run
         0 0% 90.74% 4.59MB 3.37% github.com/elastic/beats/v7/filebeat/input/v2/compat.composeFactory.Create
         0 0% 90.74% 4.09MB 3.00% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).GetStates (inline)
         0 0% 90.74% 58.14MB 42.59% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Run
         0 0% 90.74% 1.05MB 0.77% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start
         0 0% 90.74% 58.14MB 42.59% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start.func1
         0 0% 90.74% 53.11MB 38.90% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).commitStateUpdates
         0 0% 90.74% 1.05MB 0.77% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).loadStates
         0 0% 90.74% 5.03MB 3.69% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).onEvents
         0 0% 90.74% 5.03MB 3.69% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).processEventStates
         0 0% 90.74% 4.05MB 2.97% github.com/elastic/beats/v7/filebeat/registrar.New
         0 0% 90.74% 5.14MB 3.76% github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).worker
         0 0% 90.74% 7.50MB 5.50% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*Provider).publish
         0 0% 90.74% 4MB 2.93% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnAdd
         0 0% 90.74% 2.50MB 1.83% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnDelete
         0 0% 90.74% 9MB 6.59% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnUpdate
         0 0% 90.74% 4.50MB 3.30% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).containerPodEvents
         0 0% 90.74% 15.50MB 11.36% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).emit
         0 0% 90.74% 2.50MB 1.83% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).podEvent
         0 0% 90.74% 7.50MB 5.50% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).publishAll
         0 0% 90.74% 9MB 6.59% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).unlockedUpdate
         0 0% 90.74% 2MB 1.47% github.com/elastic/beats/v7/libbeat/beat.(*Event).DeepUpdateNoOverwrite (inline)
         0 0% 90.74% 2MB 1.47% github.com/elastic/beats/v7/libbeat/beat.(*Event).deepUpdate
         0 0% 90.74% 4.59MB 3.37% github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Reload
         0 0% 90.74% 4.59MB 3.37% github.com/elastic/beats/v7/libbeat/cfgfile.createRunner
         0 0% 90.74% 4.59MB 3.37% github.com/elastic/beats/v7/libbeat/cfgfile.multiplexedFactory.Create
         0 0% 90.74% 5.11MB 3.74% github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1
         0 0% 90.74% 5.11MB 3.74% github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch
         0 0% 90.74% 5.11MB 3.74% github.com/elastic/beats/v7/libbeat/cmd/instance.Run
         0 0% 90.74% 5.11MB 3.74% github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1
         0 0% 90.74% 36.51MB 26.75% github.com/elastic/beats/v7/libbeat/common/transform/typeconv.(*Converter).Convert
         0 0% 90.74% 36.51MB 26.75% github.com/elastic/beats/v7/libbeat/common/transform/typeconv.Convert
         0 0% 90.74% 2MB 1.47% github.com/elastic/beats/v7/libbeat/processors/actions.(*addFields).Run
         0 0% 90.74% 12.51MB 9.16% github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).Publish
         0 0% 90.74% 10.01MB 7.33% github.com/elastic/beats/v7/libbeat/publisher/processing.(*builder).Create.makeAddDynMetaProcessor.func4
         0 0% 90.74% 10.01MB 7.33% github.com/elastic/beats/v7/libbeat/publisher/processing.(*builder).Create.makeAddDynMetaProcessor.newAnnotateProcessor.func6
         0 0% 90.74% 12.01MB 8.80% github.com/elastic/beats/v7/libbeat/publisher/processing.(*group).Run
         0 0% 90.74% 10.01MB 7.33% github.com/elastic/beats/v7/libbeat/publisher/processing.(*processorFn).Run
         0 0% 90.74% 4.05MB 2.97% github.com/elastic/beats/v7/libbeat/statestore.(*Registry).Get
         0 0% 90.74% 44.51MB 32.61% github.com/elastic/beats/v7/libbeat/statestore.(*Store).Set
         0 0% 90.74% 4.05MB 2.97% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*Registry).Access
         0 0% 90.74% 8MB 5.86% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*diskstore).LogOperation
         0 0% 90.74% 5.50MB 4.03% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*jsonEncoder).Encode (inline)
         0 0% 90.74% 2.50MB 1.83% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*jsonEncoder).reset
         0 0% 90.74% 44.51MB 32.61% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).Set
         0 0% 90.74% 8MB 5.86% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).logOperation
         0 0% 90.74% 4.05MB 2.97% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.loadDataFile
         0 0% 90.74% 2.50MB 1.83% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.newJSONEncoder (inline)
         0 0% 90.74% 4.05MB 2.97% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.openStore
         0 0% 90.74% 4.05MB 2.97% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.readDataFile
         0 0% 90.74% 15.50MB 11.36% github.com/elastic/elastic-agent-autodiscover/kubernetes.(*watcher).Start.func1
         0 0% 90.74% 15.50MB 11.36% github.com/elastic/elastic-agent-autodiscover/kubernetes.(*watcher).process
         0 0% 90.74% 1MB 0.73% github.com/elastic/elastic-agent-autodiscover/kubernetes.PodLabels
         0 0% 90.74% 2.50MB 1.83% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*node).GenerateFromName
         0 0% 90.74% 2.50MB 1.83% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*node).GenerateK8s
         0 0% 90.74% 5MB 3.66% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*pod).Generate
         0 0% 90.74% 5MB 3.66% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*pod).GenerateK8s
         0 0% 90.74% 1.50MB 1.10% github.com/elastic/elastic-agent-autodiscover/utils.DeDot (inline)
         0 0% 90.74% 2MB 1.47% github.com/elastic/elastic-agent-libs/mapstr.M.DeepUpdateNoOverwrite (inline)
         0 0% 90.74% 42.01MB 30.78% github.com/elastic/go-structform/gotype.(*Iterator).Fold (inline)
         0 0% 90.74% 25.01MB 18.32% github.com/elastic/go-structform/gotype.(*unfoldCtx).OnString
         0 0% 90.74% 11.50MB 8.43% github.com/elastic/go-structform/gotype.(*unfoldCtx).OnUint64
         0 0% 90.74% 4.50MB 3.30% github.com/elastic/go-structform/gotype.buildFieldFold
         0 0% 90.74% 36.51MB 26.75% github.com/elastic/go-structform/gotype.buildFieldFold.makeFieldFold.func1
         0 0% 90.74% 42.01MB 30.78% github.com/elastic/go-structform/gotype.foldAnyReflect
         0 0% 90.74% 42.01MB 30.78% github.com/elastic/go-structform/gotype.foldInterfaceValue
         0 0% 90.74% 2MB 1.47% github.com/elastic/go-structform/gotype.getFoldPointer
         0 0% 90.74% 5.50MB 4.03% github.com/elastic/go-structform/gotype.getReflectFold
         0 0% 90.74% 5.50MB 4.03% github.com/elastic/go-structform/gotype.getReflectFoldStruct
         0 0% 90.74% 36.51MB 26.75% github.com/elastic/go-structform/gotype.getReflectFoldStruct.makeStructFold.func2
         0 0% 90.74% 36.51MB 26.75% github.com/elastic/go-structform/gotype.getReflectFoldStruct.makeStructFold.makeFieldsFold.func3
         0 0% 90.74% 25.01MB 18.32% github.com/elastic/go-structform/gotype.reFoldString
         0 0% 90.74% 11.50MB 8.43% github.com/elastic/go-structform/gotype.reFoldUint64
         0 0% 90.74% 5.11MB 3.74% github.com/spf13/cobra.(*Command).Execute
         0 0% 90.74% 5.11MB 3.74% github.com/spf13/cobra.(*Command).ExecuteC
         0 0% 90.74% 5.11MB 3.74% github.com/spf13/cobra.(*Command).execute
         0 0% 90.74% 1.01MB 0.74% google.golang.org/protobuf/internal/filedesc.Builder.Build
         0 0% 90.74% 1.01MB 0.74% google.golang.org/protobuf/internal/filedesc.newRawFile
         0 0% 90.74% 1.53MB 1.12% google.golang.org/protobuf/internal/filetype.Builder.Build
         0 0% 90.74% 3.50MB 2.57% k8s.io/apimachinery/pkg/runtime.(*Scheme).New
         0 0% 90.74% 20.01MB 14.66% k8s.io/apimachinery/pkg/runtime.Decode (inline)
         0 0% 90.74% 3.50MB 2.57% k8s.io/apimachinery/pkg/runtime.UseOrCreateObject
         0 0% 90.74% 21.01MB 15.39% k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode
         0 0% 90.74% 21.52MB 15.76% k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode
         0 0% 90.74% 18.01MB 13.19% k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).unmarshal
         0 0% 90.74% 16MB 11.72% k8s.io/apimachinery/pkg/util/wait.BackoffUntil
         0 0% 90.74% 16MB 11.72% k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
         0 0% 90.74% 16MB 11.72% k8s.io/apimachinery/pkg/util/wait.JitterUntil
         0 0% 90.74% 16MB 11.72% k8s.io/apimachinery/pkg/util/wait.Until
         0 0% 90.74% 19.51MB 14.29% k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive
         0 0% 90.74% 1MB 0.73% k8s.io/client-go/kubernetes/typed/core/v1.(*pods).List
         0 0% 90.74% 1MB 0.73% k8s.io/client-go/rest.Result.Into
         0 0% 90.74% 20.01MB 14.66% k8s.io/client-go/rest/watch.(*Decoder).Decode
         0 0% 90.74% 5.61MB 4.11% main.main
         0 0% 90.74% 11.51MB 8.43% reflect.MakeSlice
         0 0% 90.74% 15.60MB 11.42% runtime.doInit (inline)
         0 0% 90.74% 15.60MB 11.42% runtime.doInit1
         0 0% 90.74% 21.20MB 15.53% runtime.main
         0 0% 90.74% 18.01MB 13.19% sigs.k8s.io/json.UnmarshalCaseSensitivePreserveInts (inline)
         0 0% 90.74% 14.01MB 10.26% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).array
         0 0% 90.74% 18.01MB 13.19% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).object
         0 0% 90.74% 18.01MB 13.19% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).unmarshal
         0 0% 90.74% 18.01MB 13.19% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).value
         0 0% 90.74% 18.01MB 13.19% sigs.k8s.io/json/internal/golang/encoding/json.Unmarshal
         0 0% 90.74% 1MB 0.73% sigs.k8s.io/json/internal/golang/encoding/json.indirect
         0 0% 90.74% 2MB 1.47% strings.(*Builder).Grow
         0 0% 90.74% 1.50MB 1.10% strings.Replace

```