Filebeat memory keeps increasing until OOM

Elastic Stack Beats
1

Hello all

We are facing persistent Out of Memory(OOM) issue with filebeat across our kubernetes environment. We initially encountered the issue with Filebeat version 8.5.1. Follwing the suggestions from a topic(Filebeat: "memory leak" via filebeat.autodiscover and >>200.000 goroutines) on your forum regarding a memory leak via Filebeat autodiscover, we tested versions 8.9.0 and 8.15.0 as well. Unfortunately, the issue persistes across all these version.

Could you please help to take a look?
Backgound:

  1. Filebeat version: 8.5.1/8.9.0/8.15.0

  2. Filebeat config file:

       http.pprof.enabled: true
   filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         templates:
           - condition:
               equals:
                 kubernetes.namespace: test-namespace
             config:
               - type: container
                 paths:
                   - /var/log/containers/*-${data.kubernetes.container.id}.log
                 exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines
   multiline.pattern: '^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3})'
   multiline.negate: true
   multiline.match: after
   processors:
     - drop_fields:
         fields: ["ecs", "agent", "input", "host", "stream", "log"]
         ignore_missing: false
   output.logstash:
     enabled: true
     hosts: 'logstash:5044'

  3. 200 cronjob simulation in parallel
    In order to simulate the issue, we manually create 200 cronjob in parallel to the namespace where filebeat is gathering log.

       cronjob-teamplate.yml
   apiVersion: batch/v1
   kind: CronJob
   metadata:
     name: cronjob-XXX
     namespace: test-namespace
   spec:
     schedule: "* * * * *"
     jobTemplate:
       spec:
         template:
           spec:
             containers:
               - name: hello
                 image: busybox
                 imagePullPolicy: IfNotPresent
                 command:
                   - /bin/sh
                   - -c
                   - date; echo Hello from the Kubernetes cluster; sleep 5; exit 0
       restartPolicy: OnFailure

  4. Please find the below graph
    We have deployed filebeat on three Kubernete nodes and each instance consistently encounters OOM problems. Each instance has the similar graph as below:

    image
    image1536×341 37.8 KB

  5. Gather the heap pprof for your help
    In order to determine the root cause ASAP. We gathered the heap by pprof and pickup two bin file. One is picked up at the start of the test while second one is picked up just before OOM.
    11：25

File: filebeat
Build ID: 0685fb39bd17876b71744aebb3b7fd7fd0dd242f
Type: inuse_space
Time: Sep 2, 2024 at 11:29am (JST)
Showing nodes accounting for 99.93MB, 89.20% of 112.03MB total
Dropped 93 nodes (cum <= 0.56MB)
      flat  flat%   sum%        cum   cum%
   26.51MB 23.66% 23.66%    26.51MB 23.66%  github.com/elastic/go-structform/gotype.(*unfolderMapIfc).put (inline)
   16.01MB 14.29% 37.95%    16.01MB 14.29%  github.com/elastic/elastic-agent-libs/mapstr.M.Clone
   11.51MB 10.27% 48.22%    11.51MB 10.27%  reflect.unsafe_NewArray
    7.78MB  6.95% 55.17%     7.78MB  6.95%  github.com/elastic/beats/v7/filebeat/input/file.(*States).GetStates
       6MB  5.36% 60.53%        6MB  5.36%  github.com/goccy/go-json/internal/decoder.init.0
       4MB  3.57% 64.10%        4MB  3.57%  github.com/elastic/beats/v7/filebeat/registrar.readStatesFrom.func1
       4MB  3.57% 67.67%    36.01MB 32.14%  github.com/elastic/beats/v7/filebeat/registrar.writeStates
    3.01MB  2.68% 70.36%     3.01MB  2.68%  github.com/goccy/go-json/internal/encoder.init.0
       3MB  2.68% 73.03%        3MB  2.68%  sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).literalStore
    2.50MB  2.23% 75.27%     2.50MB  2.23%  reflect.New
    2.50MB  2.23% 77.50%     2.50MB  2.23%  encoding/json.(*decodeState).literalStore
       2MB  1.79% 79.28%        2MB  1.79%  k8s.io/apimachinery/pkg/apis/meta/v1.(*FieldsV1).UnmarshalJSON
       2MB  1.79% 81.07%        2MB  1.79%  github.com/elastic/go-structform/gotype.makeFieldFold (inline)
    1.50MB  1.34% 82.41%    18.01MB 16.07%  github.com/elastic/go-structform/gotype.(*unfolderMapIfc).OnString
    1.05MB  0.94% 83.35%     1.05MB  0.94%  github.com/elastic/beats/v7/filebeat/input/file.(*States).SetStates
    1.05MB  0.94% 84.29%     1.05MB  0.94%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.loadDataFile.func1
       1MB  0.89% 85.18%        1MB  0.89%  github.com/elastic/elastic-agent-libs/mapstr.M.Put
       1MB  0.89% 86.07%    12.01MB 10.72%  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).publish
       1MB  0.89% 86.97%        1MB  0.89%  strings.(*Builder).grow
    0.50MB  0.45% 87.41%     1.50MB  1.34%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).containerPodEvents
    0.50MB  0.45% 87.86%        3MB  2.68%  github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*Resource).GenerateK8s
    0.50MB  0.45% 88.31%        1MB  0.89%  github.com/elastic/go-structform/gotype.NewIterator
    0.50MB  0.45% 88.75%        3MB  2.68%  encoding/json.(*decodeState).object
    0.50MB  0.45% 89.20%     2.50MB  2.23%  github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.GenerateMap
         0     0% 89.20%        3MB  2.68%  encoding/json.(*Decoder).Decode
         0     0% 89.20%        3MB  2.68%  encoding/json.(*decodeState).array
         0     0% 89.20%        3MB  2.68%  encoding/json.(*decodeState).unmarshal
         0     0% 89.20%        3MB  2.68%  encoding/json.(*decodeState).value
         0     0% 89.20%     9.11MB  8.13%  github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run
         0     0% 89.20%    12.01MB 10.72%  github.com/elastic/beats/v7/filebeat/beater.(*countingClient).Publish
         0     0% 89.20%     4.05MB  3.62%  github.com/elastic/beats/v7/filebeat/beater.(*filebeatStore).Access
         0     0% 89.20%     3.89MB  3.47%  github.com/elastic/beats/v7/filebeat/channel.(*onCreateFactory).CheckConfig
         0     0% 89.20%    12.01MB 10.72%  github.com/elastic/beats/v7/filebeat/channel.(*outlet).OnEvent
         0     0% 89.20%    12.01MB 10.72%  github.com/elastic/beats/v7/filebeat/channel.SubOutlet.func1
         0     0% 89.20%     3.89MB  3.47%  github.com/elastic/beats/v7/filebeat/input.(*RunnerFactory).CheckConfig
         0     0% 89.20%     3.89MB  3.47%  github.com/elastic/beats/v7/filebeat/input.(*RunnerFactory).Create
         0     0% 89.20%     3.89MB  3.47%  github.com/elastic/beats/v7/filebeat/input/v2/compat.composeFactory.CheckConfig
         0     0% 89.20%     3.89MB  3.47%  github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).GetStates (inline)
         0     0% 89.20%    39.90MB 35.62%  github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Run
         0     0% 89.20%     5.05MB  4.51%  github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start
         0     0% 89.20%    39.90MB 35.62%  github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start.func1
         0     0% 89.20%    39.90MB 35.62%  github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).commitStateUpdates
         0     0% 89.20%     5.05MB  4.51%  github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).loadStates
         0     0% 89.20%     4.05MB  3.62%  github.com/elastic/beats/v7/filebeat/registrar.New
         0     0% 89.20%        4MB  3.57%  github.com/elastic/beats/v7/filebeat/registrar.readStatesFrom
         0     0% 89.20%     3.89MB  3.47%  github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).handleStart
         0     0% 89.20%     3.89MB  3.47%  github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).worker
         0     0% 89.20%        5MB  4.46%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*Provider).publish
         0     0% 89.20%        1MB  0.89%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnAdd
         0     0% 89.20%     1.50MB  1.34%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnDelete
         0     0% 89.20%     6.50MB  5.80%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnUpdate
         0     0% 89.20%        9MB  8.04%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).emit
         0     0% 89.20%     2.50MB  2.23%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).podEvent
         0     0% 89.20%        5MB  4.46%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).publishAll
         0     0% 89.20%     6.50MB  5.80%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).unlockedUpdate
         0     0% 89.20%     3.89MB  3.47%  github.com/elastic/beats/v7/libbeat/cfgfile.multiplexedFactory.CheckConfig
         0     0% 89.20%     9.11MB  8.13%  github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1
         0     0% 89.20%     9.11MB  8.13%  github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch
         0     0% 89.20%     9.11MB  8.13%  github.com/elastic/beats/v7/libbeat/cmd/instance.Run
         0     0% 89.20%     9.11MB  8.13%  github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1
         0     0% 89.20%    28.01MB 25.00%  github.com/elastic/beats/v7/libbeat/common/transform/typeconv.(*Converter).Convert
         0     0% 89.20%    28.01MB 25.00%  github.com/elastic/beats/v7/libbeat/common/transform/typeconv.Convert
         0     0% 89.20%    12.01MB 10.72%  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).Publish
         0     0% 89.20%    10.51MB  9.38%  github.com/elastic/beats/v7/libbeat/publisher/processing.(*builder).Create.makeAddDynMetaProcessor.func4
         0     0% 89.20%    10.51MB  9.38%  github.com/elastic/beats/v7/libbeat/publisher/processing.(*builder).Create.makeAddDynMetaProcessor.newAnnotateProcessor.func6
         0     0% 89.20%    11.01MB  9.82%  github.com/elastic/beats/v7/libbeat/publisher/processing.(*group).Run
         0     0% 89.20%    10.51MB  9.38%  github.com/elastic/beats/v7/libbeat/publisher/processing.(*processorFn).Run
         0     0% 89.20%     4.05MB  3.62%  github.com/elastic/beats/v7/libbeat/statestore.(*Registry).Get
         0     0% 89.20%        4MB  3.57%  github.com/elastic/beats/v7/libbeat/statestore.(*Store).Each
         0     0% 89.20%    32.01MB 28.57%  github.com/elastic/beats/v7/libbeat/statestore.(*Store).Set
         0     0% 89.20%     4.05MB  3.62%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*Registry).Access
         0     0% 89.20%        4MB  3.57%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*diskstore).LogOperation
         0     0% 89.20%        3MB  2.68%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*jsonEncoder).Encode (inline)
         0     0% 89.20%        1MB  0.89%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*jsonEncoder).reset
         0     0% 89.20%        4MB  3.57%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).Each
         0     0% 89.20%    32.01MB 28.57%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).Set
         0     0% 89.20%        4MB  3.57%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).logOperation
         0     0% 89.20%     4.05MB  3.62%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.loadDataFile
         0     0% 89.20%        1MB  0.89%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.newJSONEncoder (inline)
         0     0% 89.20%     4.05MB  3.62%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.openStore
         0     0% 89.20%     4.05MB  3.62%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.readDataFile
         0     0% 89.20%        9MB  8.04%  github.com/elastic/elastic-agent-autodiscover/kubernetes.(*watcher).Start.func1
         0     0% 89.20%        9MB  8.04%  github.com/elastic/elastic-agent-autodiscover/kubernetes.(*watcher).process
         0     0% 89.20%        2MB  1.79%  github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*node).GenerateFromName
         0     0% 89.20%        2MB  1.79%  github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*node).GenerateK8s
         0     0% 89.20%        3MB  2.68%  github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*pod).Generate
         0     0% 89.20%        3MB  2.68%  github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*pod).GenerateK8s
         0     0% 89.20%        1MB  0.89%  github.com/elastic/elastic-agent-autodiscover/utils.DeDot (inline)
         0     0% 89.20%    31.01MB 27.68%  github.com/elastic/go-structform/gotype.(*Iterator).Fold (inline)
         0     0% 89.20%    18.01MB 16.07%  github.com/elastic/go-structform/gotype.(*unfoldCtx).OnString
         0     0% 89.20%       10MB  8.93%  github.com/elastic/go-structform/gotype.(*unfoldCtx).OnUint64
         0     0% 89.20%       10MB  8.93%  github.com/elastic/go-structform/gotype.(*unfolderMapIfc).OnUint64
         0     0% 89.20%     2.50MB  2.23%  github.com/elastic/go-structform/gotype.buildFieldFold
         0     0% 89.20%    28.01MB 25.00%  github.com/elastic/go-structform/gotype.buildFieldFold.makeFieldFold.func1
         0     0% 89.20%    31.01MB 27.68%  github.com/elastic/go-structform/gotype.foldAnyReflect
         0     0% 89.20%    31.01MB 27.68%  github.com/elastic/go-structform/gotype.foldInterfaceValue
         0     0% 89.20%        3MB  2.68%  github.com/elastic/go-structform/gotype.getReflectFold
         0     0% 89.20%        3MB  2.68%  github.com/elastic/go-structform/gotype.getReflectFoldStruct
         0     0% 89.20%    28.01MB 25.00%  github.com/elastic/go-structform/gotype.getReflectFoldStruct.makeStructFold.func2
         0     0% 89.20%    28.01MB 25.00%  github.com/elastic/go-structform/gotype.getReflectFoldStruct.makeStructFold.makeFieldsFold.func3
         0     0% 89.20%     2.50MB  2.23%  github.com/elastic/go-structform/gotype.getStructFieldsFolds
         0     0% 89.20%    18.01MB 16.07%  github.com/elastic/go-structform/gotype.reFoldString
         0     0% 89.20%       10MB  8.93%  github.com/elastic/go-structform/gotype.reFoldUint64
         0     0% 89.20%     9.11MB  8.13%  github.com/spf13/cobra.(*Command).Execute
         0     0% 89.20%     9.11MB  8.13%  github.com/spf13/cobra.(*Command).ExecuteC
         0     0% 89.20%     9.11MB  8.13%  github.com/spf13/cobra.(*Command).execute
         0     0% 89.20%     1.01MB   0.9%  google.golang.org/protobuf/internal/filedesc.Builder.Build
         0     0% 89.20%     1.01MB   0.9%  google.golang.org/protobuf/internal/filedesc.newRawFile
         0     0% 89.20%     1.53MB  1.36%  google.golang.org/protobuf/internal/filetype.Builder.Build
         0     0% 89.20%        2MB  1.79%  k8s.io/apimachinery/pkg/runtime.(*Scheme).New
         0     0% 89.20%    18.51MB 16.52%  k8s.io/apimachinery/pkg/runtime.Decode (inline)
         0     0% 89.20%        2MB  1.79%  k8s.io/apimachinery/pkg/runtime.UseOrCreateObject
         0     0% 89.20%    19.51MB 17.42%  k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode
         0     0% 89.20%    20.02MB 17.87%  k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode
         0     0% 89.20%    18.01MB 16.08%  k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).unmarshal
         0     0% 89.20%     9.50MB  8.48%  k8s.io/apimachinery/pkg/util/wait.BackoffUntil
         0     0% 89.20%     9.50MB  8.48%  k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
         0     0% 89.20%     9.50MB  8.48%  k8s.io/apimachinery/pkg/util/wait.JitterUntil
         0     0% 89.20%     9.50MB  8.48%  k8s.io/apimachinery/pkg/util/wait.Until
         0     0% 89.20%    18.51MB 16.52%  k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive
         0     0% 89.20%        1MB  0.89%  k8s.io/client-go/kubernetes/typed/core/v1.(*pods).List
         0     0% 89.20%        1MB  0.89%  k8s.io/client-go/rest.Result.Into
         0     0% 89.20%    18.51MB 16.52%  k8s.io/client-go/rest/watch.(*Decoder).Decode
         0     0% 89.20%     9.61MB  8.58%  main.main
         0     0% 89.20%    11.51MB 10.27%  reflect.MakeSlice
         0     0% 89.20%    15.60MB 13.92%  runtime.doInit (inline)
         0     0% 89.20%    15.60MB 13.92%  runtime.doInit1
         0     0% 89.20%    25.20MB 22.50%  runtime.main
         0     0% 89.20%    18.01MB 16.08%  sigs.k8s.io/json.UnmarshalCaseSensitivePreserveInts (inline)
         0     0% 89.20%    16.51MB 14.74%  sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).array
         0     0% 89.20%    18.01MB 16.08%  sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).object
         0     0% 89.20%    18.01MB 16.08%  sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).unmarshal
         0     0% 89.20%    18.01MB 16.08%  sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).value
         0     0% 89.20%    18.01MB 16.08%  sigs.k8s.io/json/internal/golang/encoding/json.Unmarshal
         0     0% 89.20%        1MB  0.89%  strings.(*Builder).Grow
         0     0% 89.20%        1MB  0.89%  strings.Replace

14：04

File: filebeat
Build ID: 0685fb39bd17876b71744aebb3b7fd7fd0dd242f
Type: inuse_space
Time: Sep 2, 2024 at 2:05pm (JST)
Showing nodes accounting for 123.88MB, 90.74% of 136.51MB total
Dropped 103 nodes (cum <= 0.68MB)
      flat  flat%   sum%        cum   cum%
   32.51MB 23.82% 23.82%    32.51MB 23.82%  github.com/elastic/go-structform/gotype.(*unfolderMapIfc).put (inline)
   19.51MB 14.29% 38.11%    19.51MB 14.29%  github.com/elastic/elastic-agent-libs/mapstr.M.Clone
   11.51MB  8.43% 46.54%    11.51MB  8.43%  reflect.unsafe_NewArray
    8.19MB  6.00% 52.54%     8.19MB  6.00%  github.com/elastic/beats/v7/filebeat/input/file.(*States).GetStates
       6MB  4.40% 56.93%        6MB  4.40%  github.com/goccy/go-json/internal/decoder.init.0
    5.03MB  3.69% 60.62%     5.03MB  3.69%  github.com/elastic/beats/v7/filebeat/input/file.(*States).UpdateWithTs
    4.50MB  3.30% 63.92%     4.50MB  3.30%  reflect.New
    4.50MB  3.30% 67.21%    49.01MB 35.90%  github.com/elastic/beats/v7/filebeat/registrar.writeStates
    3.50MB  2.56% 69.78%     4.50MB  3.30%  sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).literalStore
    3.50MB  2.56% 72.34%    25.01MB 18.32%  github.com/elastic/go-structform/gotype.(*unfolderMapIfc).OnString
    3.01MB  2.20% 74.55%     3.01MB  2.20%  github.com/goccy/go-json/internal/encoder.init.0
    2.50MB  1.83% 76.38%     2.50MB  1.83%  encoding/json.(*decodeState).literalStore
       2MB  1.47% 77.84%        2MB  1.47%  github.com/elastic/elastic-agent-libs/mapstr.M.deepUpdateMap
       2MB  1.47% 79.31%        5MB  3.66%  github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*Resource).GenerateK8s
       2MB  1.47% 80.77%        2MB  1.47%  github.com/elastic/go-structform/json.NewVisitor (inline)
       2MB  1.47% 82.24%        2MB  1.47%  github.com/elastic/go-structform/gotype.makeFieldFold (inline)
       2MB  1.47% 83.70%        2MB  1.47%  strings.(*Builder).grow
    1.50MB  1.10% 84.80%     1.50MB  1.10%  github.com/elastic/go-structform/gotype.(*typeFoldRegistry).set (inline)
    1.50MB  1.10% 85.90%        3MB  2.20%  github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.GenerateMap
    1.05MB  0.77% 86.67%     1.05MB  0.77%  github.com/elastic/beats/v7/filebeat/input/file.(*States).SetStates
    1.05MB  0.77% 87.44%     1.05MB  0.77%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.loadDataFile.func1
       1MB  0.73% 88.18%        1MB  0.73%  k8s.io/apimachinery/pkg/apis/meta/v1.(*FieldsV1).UnmarshalJSON
       1MB  0.73% 88.91%        1MB  0.73%  github.com/elastic/beats/v7/filebeat/input/log.(*Harvester).onMessage
    0.50MB  0.37% 89.28%        1MB  0.73%  github.com/elastic/elastic-agent-libs/safemapstr.Put (inline)
    0.50MB  0.37% 89.64%    12.51MB  9.16%  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).publish
    0.50MB  0.37% 90.01%        3MB  2.20%  encoding/json.(*decodeState).object
    0.50MB  0.37% 90.37%    11.50MB  8.43%  github.com/elastic/go-structform/gotype.(*unfolderMapIfc).OnUint64
    0.50MB  0.37% 90.74%        5MB  3.66%  github.com/elastic/go-structform/gotype.getStructFieldsFolds
         0     0% 90.74%        3MB  2.20%  encoding/json.(*Decoder).Decode
         0     0% 90.74%        3MB  2.20%  encoding/json.(*decodeState).array
         0     0% 90.74%        3MB  2.20%  encoding/json.(*decodeState).unmarshal
         0     0% 90.74%        3MB  2.20%  encoding/json.(*decodeState).value
         0     0% 90.74%     5.11MB  3.74%  github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run
         0     0% 90.74%     4.59MB  3.37%  github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run.RunnerFactoryWithCommonInputSettings.func10
         0     0% 90.74%    12.51MB  9.16%  github.com/elastic/beats/v7/filebeat/beater.(*countingClient).Publish
         0     0% 90.74%     4.05MB  2.97%  github.com/elastic/beats/v7/filebeat/beater.(*filebeatStore).Access
         0     0% 90.74%     4.59MB  3.37%  github.com/elastic/beats/v7/filebeat/channel.(*onCreateFactory).Create
         0     0% 90.74%    12.51MB  9.16%  github.com/elastic/beats/v7/filebeat/channel.(*outlet).OnEvent
         0     0% 90.74%    12.51MB  9.16%  github.com/elastic/beats/v7/filebeat/channel.SubOutlet.func1
         0     0% 90.74%        1MB  0.73%  github.com/elastic/beats/v7/filebeat/harvester.(*Registry).Start.func1
         0     0% 90.74%     4.59MB  3.37%  github.com/elastic/beats/v7/filebeat/input.(*RunnerFactory).Create
         0     0% 90.74%        1MB  0.73%  github.com/elastic/beats/v7/filebeat/input/log.(*Harvester).Run
         0     0% 90.74%     4.59MB  3.37%  github.com/elastic/beats/v7/filebeat/input/v2/compat.composeFactory.Create
         0     0% 90.74%     4.09MB  3.00%  github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).GetStates (inline)
         0     0% 90.74%    58.14MB 42.59%  github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Run
         0     0% 90.74%     1.05MB  0.77%  github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start
         0     0% 90.74%    58.14MB 42.59%  github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start.func1
         0     0% 90.74%    53.11MB 38.90%  github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).commitStateUpdates
         0     0% 90.74%     1.05MB  0.77%  github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).loadStates
         0     0% 90.74%     5.03MB  3.69%  github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).onEvents
         0     0% 90.74%     5.03MB  3.69%  github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).processEventStates
         0     0% 90.74%     4.05MB  2.97%  github.com/elastic/beats/v7/filebeat/registrar.New
         0     0% 90.74%     5.14MB  3.76%  github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).worker
         0     0% 90.74%     7.50MB  5.50%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*Provider).publish
         0     0% 90.74%        4MB  2.93%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnAdd
         0     0% 90.74%     2.50MB  1.83%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnDelete
         0     0% 90.74%        9MB  6.59%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnUpdate
         0     0% 90.74%     4.50MB  3.30%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).containerPodEvents
         0     0% 90.74%    15.50MB 11.36%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).emit
         0     0% 90.74%     2.50MB  1.83%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).podEvent
         0     0% 90.74%     7.50MB  5.50%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).publishAll
         0     0% 90.74%        9MB  6.59%  github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).unlockedUpdate
         0     0% 90.74%        2MB  1.47%  github.com/elastic/beats/v7/libbeat/beat.(*Event).DeepUpdateNoOverwrite (inline)
         0     0% 90.74%        2MB  1.47%  github.com/elastic/beats/v7/libbeat/beat.(*Event).deepUpdate
         0     0% 90.74%     4.59MB  3.37%  github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Reload
         0     0% 90.74%     4.59MB  3.37%  github.com/elastic/beats/v7/libbeat/cfgfile.createRunner
         0     0% 90.74%     4.59MB  3.37%  github.com/elastic/beats/v7/libbeat/cfgfile.multiplexedFactory.Create
         0     0% 90.74%     5.11MB  3.74%  github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1
         0     0% 90.74%     5.11MB  3.74%  github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch
         0     0% 90.74%     5.11MB  3.74%  github.com/elastic/beats/v7/libbeat/cmd/instance.Run
         0     0% 90.74%     5.11MB  3.74%  github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1
         0     0% 90.74%    36.51MB 26.75%  github.com/elastic/beats/v7/libbeat/common/transform/typeconv.(*Converter).Convert
         0     0% 90.74%    36.51MB 26.75%  github.com/elastic/beats/v7/libbeat/common/transform/typeconv.Convert
         0     0% 90.74%        2MB  1.47%  github.com/elastic/beats/v7/libbeat/processors/actions.(*addFields).Run
         0     0% 90.74%    12.51MB  9.16%  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).Publish
         0     0% 90.74%    10.01MB  7.33%  github.com/elastic/beats/v7/libbeat/publisher/processing.(*builder).Create.makeAddDynMetaProcessor.func4
         0     0% 90.74%    10.01MB  7.33%  github.com/elastic/beats/v7/libbeat/publisher/processing.(*builder).Create.makeAddDynMetaProcessor.newAnnotateProcessor.func6
         0     0% 90.74%    12.01MB  8.80%  github.com/elastic/beats/v7/libbeat/publisher/processing.(*group).Run
         0     0% 90.74%    10.01MB  7.33%  github.com/elastic/beats/v7/libbeat/publisher/processing.(*processorFn).Run
         0     0% 90.74%     4.05MB  2.97%  github.com/elastic/beats/v7/libbeat/statestore.(*Registry).Get
         0     0% 90.74%    44.51MB 32.61%  github.com/elastic/beats/v7/libbeat/statestore.(*Store).Set
         0     0% 90.74%     4.05MB  2.97%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*Registry).Access
         0     0% 90.74%        8MB  5.86%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*diskstore).LogOperation
         0     0% 90.74%     5.50MB  4.03%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*jsonEncoder).Encode (inline)
         0     0% 90.74%     2.50MB  1.83%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*jsonEncoder).reset
         0     0% 90.74%    44.51MB 32.61%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).Set
         0     0% 90.74%        8MB  5.86%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).logOperation
         0     0% 90.74%     4.05MB  2.97%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.loadDataFile
         0     0% 90.74%     2.50MB  1.83%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.newJSONEncoder (inline)
         0     0% 90.74%     4.05MB  2.97%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.openStore
         0     0% 90.74%     4.05MB  2.97%  github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.readDataFile
         0     0% 90.74%    15.50MB 11.36%  github.com/elastic/elastic-agent-autodiscover/kubernetes.(*watcher).Start.func1
         0     0% 90.74%    15.50MB 11.36%  github.com/elastic/elastic-agent-autodiscover/kubernetes.(*watcher).process
         0     0% 90.74%        1MB  0.73%  github.com/elastic/elastic-agent-autodiscover/kubernetes.PodLabels
         0     0% 90.74%     2.50MB  1.83%  github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*node).GenerateFromName
         0     0% 90.74%     2.50MB  1.83%  github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*node).GenerateK8s
         0     0% 90.74%        5MB  3.66%  github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*pod).Generate
         0     0% 90.74%        5MB  3.66%  github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*pod).GenerateK8s
         0     0% 90.74%     1.50MB  1.10%  github.com/elastic/elastic-agent-autodiscover/utils.DeDot (inline)
         0     0% 90.74%        2MB  1.47%  github.com/elastic/elastic-agent-libs/mapstr.M.DeepUpdateNoOverwrite (inline)
         0     0% 90.74%    42.01MB 30.78%  github.com/elastic/go-structform/gotype.(*Iterator).Fold (inline)
         0     0% 90.74%    25.01MB 18.32%  github.com/elastic/go-structform/gotype.(*unfoldCtx).OnString
         0     0% 90.74%    11.50MB  8.43%  github.com/elastic/go-structform/gotype.(*unfoldCtx).OnUint64
         0     0% 90.74%     4.50MB  3.30%  github.com/elastic/go-structform/gotype.buildFieldFold
         0     0% 90.74%    36.51MB 26.75%  github.com/elastic/go-structform/gotype.buildFieldFold.makeFieldFold.func1
         0     0% 90.74%    42.01MB 30.78%  github.com/elastic/go-structform/gotype.foldAnyReflect
         0     0% 90.74%    42.01MB 30.78%  github.com/elastic/go-structform/gotype.foldInterfaceValue
         0     0% 90.74%        2MB  1.47%  github.com/elastic/go-structform/gotype.getFoldPointer
         0     0% 90.74%     5.50MB  4.03%  github.com/elastic/go-structform/gotype.getReflectFold
         0     0% 90.74%     5.50MB  4.03%  github.com/elastic/go-structform/gotype.getReflectFoldStruct
         0     0% 90.74%    36.51MB 26.75%  github.com/elastic/go-structform/gotype.getReflectFoldStruct.makeStructFold.func2
         0     0% 90.74%    36.51MB 26.75%  github.com/elastic/go-structform/gotype.getReflectFoldStruct.makeStructFold.makeFieldsFold.func3
         0     0% 90.74%    25.01MB 18.32%  github.com/elastic/go-structform/gotype.reFoldString
         0     0% 90.74%    11.50MB  8.43%  github.com/elastic/go-structform/gotype.reFoldUint64
         0     0% 90.74%     5.11MB  3.74%  github.com/spf13/cobra.(*Command).Execute
         0     0% 90.74%     5.11MB  3.74%  github.com/spf13/cobra.(*Command).ExecuteC
         0     0% 90.74%     5.11MB  3.74%  github.com/spf13/cobra.(*Command).execute
         0     0% 90.74%     1.01MB  0.74%  google.golang.org/protobuf/internal/filedesc.Builder.Build
         0     0% 90.74%     1.01MB  0.74%  google.golang.org/protobuf/internal/filedesc.newRawFile
         0     0% 90.74%     1.53MB  1.12%  google.golang.org/protobuf/internal/filetype.Builder.Build
         0     0% 90.74%     3.50MB  2.57%  k8s.io/apimachinery/pkg/runtime.(*Scheme).New
         0     0% 90.74%    20.01MB 14.66%  k8s.io/apimachinery/pkg/runtime.Decode (inline)
         0     0% 90.74%     3.50MB  2.57%  k8s.io/apimachinery/pkg/runtime.UseOrCreateObject
         0     0% 90.74%    21.01MB 15.39%  k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode
         0     0% 90.74%    21.52MB 15.76%  k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode
         0     0% 90.74%    18.01MB 13.19%  k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).unmarshal
         0     0% 90.74%       16MB 11.72%  k8s.io/apimachinery/pkg/util/wait.BackoffUntil
         0     0% 90.74%       16MB 11.72%  k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
         0     0% 90.74%       16MB 11.72%  k8s.io/apimachinery/pkg/util/wait.JitterUntil
         0     0% 90.74%       16MB 11.72%  k8s.io/apimachinery/pkg/util/wait.Until
         0     0% 90.74%    19.51MB 14.29%  k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive
         0     0% 90.74%        1MB  0.73%  k8s.io/client-go/kubernetes/typed/core/v1.(*pods).List
         0     0% 90.74%        1MB  0.73%  k8s.io/client-go/rest.Result.Into
         0     0% 90.74%    20.01MB 14.66%  k8s.io/client-go/rest/watch.(*Decoder).Decode
         0     0% 90.74%     5.61MB  4.11%  main.main
         0     0% 90.74%    11.51MB  8.43%  reflect.MakeSlice
         0     0% 90.74%    15.60MB 11.42%  runtime.doInit (inline)
         0     0% 90.74%    15.60MB 11.42%  runtime.doInit1
         0     0% 90.74%    21.20MB 15.53%  runtime.main
         0     0% 90.74%    18.01MB 13.19%  sigs.k8s.io/json.UnmarshalCaseSensitivePreserveInts (inline)
         0     0% 90.74%    14.01MB 10.26%  sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).array
         0     0% 90.74%    18.01MB 13.19%  sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).object
         0     0% 90.74%    18.01MB 13.19%  sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).unmarshal
         0     0% 90.74%    18.01MB 13.19%  sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).value
         0     0% 90.74%    18.01MB 13.19%  sigs.k8s.io/json/internal/golang/encoding/json.Unmarshal
         0     0% 90.74%        1MB  0.73%  sigs.k8s.io/json/internal/golang/encoding/json.indirect
         0     0% 90.74%        2MB  1.47%  strings.(*Builder).Grow
         0     0% 90.74%     1.50MB  1.10%  strings.Replace