Hello all
We are facing persistent Out of Memory(OOM) issue with filebeat across our kubernetes environment. We initially encountered the issue with Filebeat version 8.5.1. Follwing the suggestions from a topic(Filebeat: "memory leak" via filebeat.autodiscover and >>200.000 goroutines) on your forum regarding a memory leak via Filebeat autodiscover, we tested versions 8.9.0 and 8.15.0 as well. Unfortunately, the issue persistes across all these version.
Could you please help to take a look?
Backgound:
-
Filebeat version: 8.5.1/8.9.0/8.15.0
-
Filebeat config file:
http.pprof.enabled: true filebeat.autodiscover: providers: - type: kubernetes node: ${NODE_NAME} templates: - condition: equals: kubernetes.namespace: test-namespace config: - type: container paths: - /var/log/containers/*-${data.kubernetes.container.id}.log exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart lines multiline.pattern: '^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3})' multiline.negate: true multiline.match: after processors: - drop_fields: fields: ["ecs", "agent", "input", "host", "stream", "log"] ignore_missing: false output.logstash: enabled: true hosts: 'logstash:5044' -
200 cronjob simulation in parallel
In order to simulate the issue, we manually create 200 cronjob in parallel to the namespace where filebeat is gathering log.cronjob-teamplate.yml apiVersion: batch/v1 kind: CronJob metadata: name: cronjob-XXX namespace: test-namespace spec: schedule: "* * * * *" jobTemplate: spec: template: spec: containers: - name: hello image: busybox imagePullPolicy: IfNotPresent command: - /bin/sh - -c - date; echo Hello from the Kubernetes cluster; sleep 5; exit 0 restartPolicy: OnFailure -
Please find the below graph
We have deployed filebeat on three Kubernete nodes and each instance consistently encounters OOM problems. Each instance has the similar graph as below:
-
Gather the heap pprof for your help
In order to determine the root cause ASAP. We gathered the heap by pprof and pickup two bin file. One is picked up at the start of the test while second one is picked up just before OOM.
11：25
File: filebeat
Build ID: 0685fb39bd17876b71744aebb3b7fd7fd0dd242f
Type: inuse_space
Time: Sep 2, 2024 at 11:29am (JST)
Showing nodes accounting for 99.93MB, 89.20% of 112.03MB total
Dropped 93 nodes (cum <= 0.56MB)
flat flat% sum% cum cum%
26.51MB 23.66% 23.66% 26.51MB 23.66% github.com/elastic/go-structform/gotype.(*unfolderMapIfc).put (inline)
16.01MB 14.29% 37.95% 16.01MB 14.29% github.com/elastic/elastic-agent-libs/mapstr.M.Clone
11.51MB 10.27% 48.22% 11.51MB 10.27% reflect.unsafe_NewArray
7.78MB 6.95% 55.17% 7.78MB 6.95% github.com/elastic/beats/v7/filebeat/input/file.(*States).GetStates
6MB 5.36% 60.53% 6MB 5.36% github.com/goccy/go-json/internal/decoder.init.0
4MB 3.57% 64.10% 4MB 3.57% github.com/elastic/beats/v7/filebeat/registrar.readStatesFrom.func1
4MB 3.57% 67.67% 36.01MB 32.14% github.com/elastic/beats/v7/filebeat/registrar.writeStates
3.01MB 2.68% 70.36% 3.01MB 2.68% github.com/goccy/go-json/internal/encoder.init.0
3MB 2.68% 73.03% 3MB 2.68% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).literalStore
2.50MB 2.23% 75.27% 2.50MB 2.23% reflect.New
2.50MB 2.23% 77.50% 2.50MB 2.23% encoding/json.(*decodeState).literalStore
2MB 1.79% 79.28% 2MB 1.79% k8s.io/apimachinery/pkg/apis/meta/v1.(*FieldsV1).UnmarshalJSON
2MB 1.79% 81.07% 2MB 1.79% github.com/elastic/go-structform/gotype.makeFieldFold (inline)
1.50MB 1.34% 82.41% 18.01MB 16.07% github.com/elastic/go-structform/gotype.(*unfolderMapIfc).OnString
1.05MB 0.94% 83.35% 1.05MB 0.94% github.com/elastic/beats/v7/filebeat/input/file.(*States).SetStates
1.05MB 0.94% 84.29% 1.05MB 0.94% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.loadDataFile.func1
1MB 0.89% 85.18% 1MB 0.89% github.com/elastic/elastic-agent-libs/mapstr.M.Put
1MB 0.89% 86.07% 12.01MB 10.72% github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).publish
1MB 0.89% 86.97% 1MB 0.89% strings.(*Builder).grow
0.50MB 0.45% 87.41% 1.50MB 1.34% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).containerPodEvents
0.50MB 0.45% 87.86% 3MB 2.68% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*Resource).GenerateK8s
0.50MB 0.45% 88.31% 1MB 0.89% github.com/elastic/go-structform/gotype.NewIterator
0.50MB 0.45% 88.75% 3MB 2.68% encoding/json.(*decodeState).object
0.50MB 0.45% 89.20% 2.50MB 2.23% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.GenerateMap
0 0% 89.20% 3MB 2.68% encoding/json.(*Decoder).Decode
0 0% 89.20% 3MB 2.68% encoding/json.(*decodeState).array
0 0% 89.20% 3MB 2.68% encoding/json.(*decodeState).unmarshal
0 0% 89.20% 3MB 2.68% encoding/json.(*decodeState).value
0 0% 89.20% 9.11MB 8.13% github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run
0 0% 89.20% 12.01MB 10.72% github.com/elastic/beats/v7/filebeat/beater.(*countingClient).Publish
0 0% 89.20% 4.05MB 3.62% github.com/elastic/beats/v7/filebeat/beater.(*filebeatStore).Access
0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/filebeat/channel.(*onCreateFactory).CheckConfig
0 0% 89.20% 12.01MB 10.72% github.com/elastic/beats/v7/filebeat/channel.(*outlet).OnEvent
0 0% 89.20% 12.01MB 10.72% github.com/elastic/beats/v7/filebeat/channel.SubOutlet.func1
0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/filebeat/input.(*RunnerFactory).CheckConfig
0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/filebeat/input.(*RunnerFactory).Create
0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/filebeat/input/v2/compat.composeFactory.CheckConfig
0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).GetStates (inline)
0 0% 89.20% 39.90MB 35.62% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Run
0 0% 89.20% 5.05MB 4.51% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start
0 0% 89.20% 39.90MB 35.62% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start.func1
0 0% 89.20% 39.90MB 35.62% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).commitStateUpdates
0 0% 89.20% 5.05MB 4.51% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).loadStates
0 0% 89.20% 4.05MB 3.62% github.com/elastic/beats/v7/filebeat/registrar.New
0 0% 89.20% 4MB 3.57% github.com/elastic/beats/v7/filebeat/registrar.readStatesFrom
0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).handleStart
0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).worker
0 0% 89.20% 5MB 4.46% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*Provider).publish
0 0% 89.20% 1MB 0.89% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnAdd
0 0% 89.20% 1.50MB 1.34% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnDelete
0 0% 89.20% 6.50MB 5.80% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnUpdate
0 0% 89.20% 9MB 8.04% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).emit
0 0% 89.20% 2.50MB 2.23% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).podEvent
0 0% 89.20% 5MB 4.46% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).publishAll
0 0% 89.20% 6.50MB 5.80% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).unlockedUpdate
0 0% 89.20% 3.89MB 3.47% github.com/elastic/beats/v7/libbeat/cfgfile.multiplexedFactory.CheckConfig
0 0% 89.20% 9.11MB 8.13% github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1
0 0% 89.20% 9.11MB 8.13% github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch
0 0% 89.20% 9.11MB 8.13% github.com/elastic/beats/v7/libbeat/cmd/instance.Run
0 0% 89.20% 9.11MB 8.13% github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1
0 0% 89.20% 28.01MB 25.00% github.com/elastic/beats/v7/libbeat/common/transform/typeconv.(*Converter).Convert
0 0% 89.20% 28.01MB 25.00% github.com/elastic/beats/v7/libbeat/common/transform/typeconv.Convert
0 0% 89.20% 12.01MB 10.72% github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).Publish
0 0% 89.20% 10.51MB 9.38% github.com/elastic/beats/v7/libbeat/publisher/processing.(*builder).Create.makeAddDynMetaProcessor.func4
0 0% 89.20% 10.51MB 9.38% github.com/elastic/beats/v7/libbeat/publisher/processing.(*builder).Create.makeAddDynMetaProcessor.newAnnotateProcessor.func6
0 0% 89.20% 11.01MB 9.82% github.com/elastic/beats/v7/libbeat/publisher/processing.(*group).Run
0 0% 89.20% 10.51MB 9.38% github.com/elastic/beats/v7/libbeat/publisher/processing.(*processorFn).Run
0 0% 89.20% 4.05MB 3.62% github.com/elastic/beats/v7/libbeat/statestore.(*Registry).Get
0 0% 89.20% 4MB 3.57% github.com/elastic/beats/v7/libbeat/statestore.(*Store).Each
0 0% 89.20% 32.01MB 28.57% github.com/elastic/beats/v7/libbeat/statestore.(*Store).Set
0 0% 89.20% 4.05MB 3.62% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*Registry).Access
0 0% 89.20% 4MB 3.57% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*diskstore).LogOperation
0 0% 89.20% 3MB 2.68% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*jsonEncoder).Encode (inline)
0 0% 89.20% 1MB 0.89% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*jsonEncoder).reset
0 0% 89.20% 4MB 3.57% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).Each
0 0% 89.20% 32.01MB 28.57% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).Set
0 0% 89.20% 4MB 3.57% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).logOperation
0 0% 89.20% 4.05MB 3.62% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.loadDataFile
0 0% 89.20% 1MB 0.89% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.newJSONEncoder (inline)
0 0% 89.20% 4.05MB 3.62% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.openStore
0 0% 89.20% 4.05MB 3.62% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.readDataFile
0 0% 89.20% 9MB 8.04% github.com/elastic/elastic-agent-autodiscover/kubernetes.(*watcher).Start.func1
0 0% 89.20% 9MB 8.04% github.com/elastic/elastic-agent-autodiscover/kubernetes.(*watcher).process
0 0% 89.20% 2MB 1.79% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*node).GenerateFromName
0 0% 89.20% 2MB 1.79% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*node).GenerateK8s
0 0% 89.20% 3MB 2.68% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*pod).Generate
0 0% 89.20% 3MB 2.68% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*pod).GenerateK8s
0 0% 89.20% 1MB 0.89% github.com/elastic/elastic-agent-autodiscover/utils.DeDot (inline)
0 0% 89.20% 31.01MB 27.68% github.com/elastic/go-structform/gotype.(*Iterator).Fold (inline)
0 0% 89.20% 18.01MB 16.07% github.com/elastic/go-structform/gotype.(*unfoldCtx).OnString
0 0% 89.20% 10MB 8.93% github.com/elastic/go-structform/gotype.(*unfoldCtx).OnUint64
0 0% 89.20% 10MB 8.93% github.com/elastic/go-structform/gotype.(*unfolderMapIfc).OnUint64
0 0% 89.20% 2.50MB 2.23% github.com/elastic/go-structform/gotype.buildFieldFold
0 0% 89.20% 28.01MB 25.00% github.com/elastic/go-structform/gotype.buildFieldFold.makeFieldFold.func1
0 0% 89.20% 31.01MB 27.68% github.com/elastic/go-structform/gotype.foldAnyReflect
0 0% 89.20% 31.01MB 27.68% github.com/elastic/go-structform/gotype.foldInterfaceValue
0 0% 89.20% 3MB 2.68% github.com/elastic/go-structform/gotype.getReflectFold
0 0% 89.20% 3MB 2.68% github.com/elastic/go-structform/gotype.getReflectFoldStruct
0 0% 89.20% 28.01MB 25.00% github.com/elastic/go-structform/gotype.getReflectFoldStruct.makeStructFold.func2
0 0% 89.20% 28.01MB 25.00% github.com/elastic/go-structform/gotype.getReflectFoldStruct.makeStructFold.makeFieldsFold.func3
0 0% 89.20% 2.50MB 2.23% github.com/elastic/go-structform/gotype.getStructFieldsFolds
0 0% 89.20% 18.01MB 16.07% github.com/elastic/go-structform/gotype.reFoldString
0 0% 89.20% 10MB 8.93% github.com/elastic/go-structform/gotype.reFoldUint64
0 0% 89.20% 9.11MB 8.13% github.com/spf13/cobra.(*Command).Execute
0 0% 89.20% 9.11MB 8.13% github.com/spf13/cobra.(*Command).ExecuteC
0 0% 89.20% 9.11MB 8.13% github.com/spf13/cobra.(*Command).execute
0 0% 89.20% 1.01MB 0.9% google.golang.org/protobuf/internal/filedesc.Builder.Build
0 0% 89.20% 1.01MB 0.9% google.golang.org/protobuf/internal/filedesc.newRawFile
0 0% 89.20% 1.53MB 1.36% google.golang.org/protobuf/internal/filetype.Builder.Build
0 0% 89.20% 2MB 1.79% k8s.io/apimachinery/pkg/runtime.(*Scheme).New
0 0% 89.20% 18.51MB 16.52% k8s.io/apimachinery/pkg/runtime.Decode (inline)
0 0% 89.20% 2MB 1.79% k8s.io/apimachinery/pkg/runtime.UseOrCreateObject
0 0% 89.20% 19.51MB 17.42% k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode
0 0% 89.20% 20.02MB 17.87% k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode
0 0% 89.20% 18.01MB 16.08% k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).unmarshal
0 0% 89.20% 9.50MB 8.48% k8s.io/apimachinery/pkg/util/wait.BackoffUntil
0 0% 89.20% 9.50MB 8.48% k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
0 0% 89.20% 9.50MB 8.48% k8s.io/apimachinery/pkg/util/wait.JitterUntil
0 0% 89.20% 9.50MB 8.48% k8s.io/apimachinery/pkg/util/wait.Until
0 0% 89.20% 18.51MB 16.52% k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive
0 0% 89.20% 1MB 0.89% k8s.io/client-go/kubernetes/typed/core/v1.(*pods).List
0 0% 89.20% 1MB 0.89% k8s.io/client-go/rest.Result.Into
0 0% 89.20% 18.51MB 16.52% k8s.io/client-go/rest/watch.(*Decoder).Decode
0 0% 89.20% 9.61MB 8.58% main.main
0 0% 89.20% 11.51MB 10.27% reflect.MakeSlice
0 0% 89.20% 15.60MB 13.92% runtime.doInit (inline)
0 0% 89.20% 15.60MB 13.92% runtime.doInit1
0 0% 89.20% 25.20MB 22.50% runtime.main
0 0% 89.20% 18.01MB 16.08% sigs.k8s.io/json.UnmarshalCaseSensitivePreserveInts (inline)
0 0% 89.20% 16.51MB 14.74% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).array
0 0% 89.20% 18.01MB 16.08% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).object
0 0% 89.20% 18.01MB 16.08% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).unmarshal
0 0% 89.20% 18.01MB 16.08% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).value
0 0% 89.20% 18.01MB 16.08% sigs.k8s.io/json/internal/golang/encoding/json.Unmarshal
0 0% 89.20% 1MB 0.89% strings.(*Builder).Grow
0 0% 89.20% 1MB 0.89% strings.Replace
14：04
File: filebeat
Build ID: 0685fb39bd17876b71744aebb3b7fd7fd0dd242f
Type: inuse_space
Time: Sep 2, 2024 at 2:05pm (JST)
Showing nodes accounting for 123.88MB, 90.74% of 136.51MB total
Dropped 103 nodes (cum <= 0.68MB)
flat flat% sum% cum cum%
32.51MB 23.82% 23.82% 32.51MB 23.82% github.com/elastic/go-structform/gotype.(*unfolderMapIfc).put (inline)
19.51MB 14.29% 38.11% 19.51MB 14.29% github.com/elastic/elastic-agent-libs/mapstr.M.Clone
11.51MB 8.43% 46.54% 11.51MB 8.43% reflect.unsafe_NewArray
8.19MB 6.00% 52.54% 8.19MB 6.00% github.com/elastic/beats/v7/filebeat/input/file.(*States).GetStates
6MB 4.40% 56.93% 6MB 4.40% github.com/goccy/go-json/internal/decoder.init.0
5.03MB 3.69% 60.62% 5.03MB 3.69% github.com/elastic/beats/v7/filebeat/input/file.(*States).UpdateWithTs
4.50MB 3.30% 63.92% 4.50MB 3.30% reflect.New
4.50MB 3.30% 67.21% 49.01MB 35.90% github.com/elastic/beats/v7/filebeat/registrar.writeStates
3.50MB 2.56% 69.78% 4.50MB 3.30% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).literalStore
3.50MB 2.56% 72.34% 25.01MB 18.32% github.com/elastic/go-structform/gotype.(*unfolderMapIfc).OnString
3.01MB 2.20% 74.55% 3.01MB 2.20% github.com/goccy/go-json/internal/encoder.init.0
2.50MB 1.83% 76.38% 2.50MB 1.83% encoding/json.(*decodeState).literalStore
2MB 1.47% 77.84% 2MB 1.47% github.com/elastic/elastic-agent-libs/mapstr.M.deepUpdateMap
2MB 1.47% 79.31% 5MB 3.66% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*Resource).GenerateK8s
2MB 1.47% 80.77% 2MB 1.47% github.com/elastic/go-structform/json.NewVisitor (inline)
2MB 1.47% 82.24% 2MB 1.47% github.com/elastic/go-structform/gotype.makeFieldFold (inline)
2MB 1.47% 83.70% 2MB 1.47% strings.(*Builder).grow
1.50MB 1.10% 84.80% 1.50MB 1.10% github.com/elastic/go-structform/gotype.(*typeFoldRegistry).set (inline)
1.50MB 1.10% 85.90% 3MB 2.20% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.GenerateMap
1.05MB 0.77% 86.67% 1.05MB 0.77% github.com/elastic/beats/v7/filebeat/input/file.(*States).SetStates
1.05MB 0.77% 87.44% 1.05MB 0.77% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.loadDataFile.func1
1MB 0.73% 88.18% 1MB 0.73% k8s.io/apimachinery/pkg/apis/meta/v1.(*FieldsV1).UnmarshalJSON
1MB 0.73% 88.91% 1MB 0.73% github.com/elastic/beats/v7/filebeat/input/log.(*Harvester).onMessage
0.50MB 0.37% 89.28% 1MB 0.73% github.com/elastic/elastic-agent-libs/safemapstr.Put (inline)
0.50MB 0.37% 89.64% 12.51MB 9.16% github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).publish
0.50MB 0.37% 90.01% 3MB 2.20% encoding/json.(*decodeState).object
0.50MB 0.37% 90.37% 11.50MB 8.43% github.com/elastic/go-structform/gotype.(*unfolderMapIfc).OnUint64
0.50MB 0.37% 90.74% 5MB 3.66% github.com/elastic/go-structform/gotype.getStructFieldsFolds
0 0% 90.74% 3MB 2.20% encoding/json.(*Decoder).Decode
0 0% 90.74% 3MB 2.20% encoding/json.(*decodeState).array
0 0% 90.74% 3MB 2.20% encoding/json.(*decodeState).unmarshal
0 0% 90.74% 3MB 2.20% encoding/json.(*decodeState).value
0 0% 90.74% 5.11MB 3.74% github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run
0 0% 90.74% 4.59MB 3.37% github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run.RunnerFactoryWithCommonInputSettings.func10
0 0% 90.74% 12.51MB 9.16% github.com/elastic/beats/v7/filebeat/beater.(*countingClient).Publish
0 0% 90.74% 4.05MB 2.97% github.com/elastic/beats/v7/filebeat/beater.(*filebeatStore).Access
0 0% 90.74% 4.59MB 3.37% github.com/elastic/beats/v7/filebeat/channel.(*onCreateFactory).Create
0 0% 90.74% 12.51MB 9.16% github.com/elastic/beats/v7/filebeat/channel.(*outlet).OnEvent
0 0% 90.74% 12.51MB 9.16% github.com/elastic/beats/v7/filebeat/channel.SubOutlet.func1
0 0% 90.74% 1MB 0.73% github.com/elastic/beats/v7/filebeat/harvester.(*Registry).Start.func1
0 0% 90.74% 4.59MB 3.37% github.com/elastic/beats/v7/filebeat/input.(*RunnerFactory).Create
0 0% 90.74% 1MB 0.73% github.com/elastic/beats/v7/filebeat/input/log.(*Harvester).Run
0 0% 90.74% 4.59MB 3.37% github.com/elastic/beats/v7/filebeat/input/v2/compat.composeFactory.Create
0 0% 90.74% 4.09MB 3.00% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).GetStates (inline)
0 0% 90.74% 58.14MB 42.59% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Run
0 0% 90.74% 1.05MB 0.77% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start
0 0% 90.74% 58.14MB 42.59% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start.func1
0 0% 90.74% 53.11MB 38.90% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).commitStateUpdates
0 0% 90.74% 1.05MB 0.77% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).loadStates
0 0% 90.74% 5.03MB 3.69% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).onEvents
0 0% 90.74% 5.03MB 3.69% github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).processEventStates
0 0% 90.74% 4.05MB 2.97% github.com/elastic/beats/v7/filebeat/registrar.New
0 0% 90.74% 5.14MB 3.76% github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).worker
0 0% 90.74% 7.50MB 5.50% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*Provider).publish
0 0% 90.74% 4MB 2.93% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnAdd
0 0% 90.74% 2.50MB 1.83% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnDelete
0 0% 90.74% 9MB 6.59% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).OnUpdate
0 0% 90.74% 4.50MB 3.30% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).containerPodEvents
0 0% 90.74% 15.50MB 11.36% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).emit
0 0% 90.74% 2.50MB 1.83% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).podEvent
0 0% 90.74% 7.50MB 5.50% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).publishAll
0 0% 90.74% 9MB 6.59% github.com/elastic/beats/v7/libbeat/autodiscover/providers/kubernetes.(*pod).unlockedUpdate
0 0% 90.74% 2MB 1.47% github.com/elastic/beats/v7/libbeat/beat.(*Event).DeepUpdateNoOverwrite (inline)
0 0% 90.74% 2MB 1.47% github.com/elastic/beats/v7/libbeat/beat.(*Event).deepUpdate
0 0% 90.74% 4.59MB 3.37% github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Reload
0 0% 90.74% 4.59MB 3.37% github.com/elastic/beats/v7/libbeat/cfgfile.createRunner
0 0% 90.74% 4.59MB 3.37% github.com/elastic/beats/v7/libbeat/cfgfile.multiplexedFactory.Create
0 0% 90.74% 5.11MB 3.74% github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1
0 0% 90.74% 5.11MB 3.74% github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch
0 0% 90.74% 5.11MB 3.74% github.com/elastic/beats/v7/libbeat/cmd/instance.Run
0 0% 90.74% 5.11MB 3.74% github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1
0 0% 90.74% 36.51MB 26.75% github.com/elastic/beats/v7/libbeat/common/transform/typeconv.(*Converter).Convert
0 0% 90.74% 36.51MB 26.75% github.com/elastic/beats/v7/libbeat/common/transform/typeconv.Convert
0 0% 90.74% 2MB 1.47% github.com/elastic/beats/v7/libbeat/processors/actions.(*addFields).Run
0 0% 90.74% 12.51MB 9.16% github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).Publish
0 0% 90.74% 10.01MB 7.33% github.com/elastic/beats/v7/libbeat/publisher/processing.(*builder).Create.makeAddDynMetaProcessor.func4
0 0% 90.74% 10.01MB 7.33% github.com/elastic/beats/v7/libbeat/publisher/processing.(*builder).Create.makeAddDynMetaProcessor.newAnnotateProcessor.func6
0 0% 90.74% 12.01MB 8.80% github.com/elastic/beats/v7/libbeat/publisher/processing.(*group).Run
0 0% 90.74% 10.01MB 7.33% github.com/elastic/beats/v7/libbeat/publisher/processing.(*processorFn).Run
0 0% 90.74% 4.05MB 2.97% github.com/elastic/beats/v7/libbeat/statestore.(*Registry).Get
0 0% 90.74% 44.51MB 32.61% github.com/elastic/beats/v7/libbeat/statestore.(*Store).Set
0 0% 90.74% 4.05MB 2.97% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*Registry).Access
0 0% 90.74% 8MB 5.86% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*diskstore).LogOperation
0 0% 90.74% 5.50MB 4.03% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*jsonEncoder).Encode (inline)
0 0% 90.74% 2.50MB 1.83% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*jsonEncoder).reset
0 0% 90.74% 44.51MB 32.61% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).Set
0 0% 90.74% 8MB 5.86% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.(*store).logOperation
0 0% 90.74% 4.05MB 2.97% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.loadDataFile
0 0% 90.74% 2.50MB 1.83% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.newJSONEncoder (inline)
0 0% 90.74% 4.05MB 2.97% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.openStore
0 0% 90.74% 4.05MB 2.97% github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.readDataFile
0 0% 90.74% 15.50MB 11.36% github.com/elastic/elastic-agent-autodiscover/kubernetes.(*watcher).Start.func1
0 0% 90.74% 15.50MB 11.36% github.com/elastic/elastic-agent-autodiscover/kubernetes.(*watcher).process
0 0% 90.74% 1MB 0.73% github.com/elastic/elastic-agent-autodiscover/kubernetes.PodLabels
0 0% 90.74% 2.50MB 1.83% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*node).GenerateFromName
0 0% 90.74% 2.50MB 1.83% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*node).GenerateK8s
0 0% 90.74% 5MB 3.66% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*pod).Generate
0 0% 90.74% 5MB 3.66% github.com/elastic/elastic-agent-autodiscover/kubernetes/metadata.(*pod).GenerateK8s
0 0% 90.74% 1.50MB 1.10% github.com/elastic/elastic-agent-autodiscover/utils.DeDot (inline)
0 0% 90.74% 2MB 1.47% github.com/elastic/elastic-agent-libs/mapstr.M.DeepUpdateNoOverwrite (inline)
0 0% 90.74% 42.01MB 30.78% github.com/elastic/go-structform/gotype.(*Iterator).Fold (inline)
0 0% 90.74% 25.01MB 18.32% github.com/elastic/go-structform/gotype.(*unfoldCtx).OnString
0 0% 90.74% 11.50MB 8.43% github.com/elastic/go-structform/gotype.(*unfoldCtx).OnUint64
0 0% 90.74% 4.50MB 3.30% github.com/elastic/go-structform/gotype.buildFieldFold
0 0% 90.74% 36.51MB 26.75% github.com/elastic/go-structform/gotype.buildFieldFold.makeFieldFold.func1
0 0% 90.74% 42.01MB 30.78% github.com/elastic/go-structform/gotype.foldAnyReflect
0 0% 90.74% 42.01MB 30.78% github.com/elastic/go-structform/gotype.foldInterfaceValue
0 0% 90.74% 2MB 1.47% github.com/elastic/go-structform/gotype.getFoldPointer
0 0% 90.74% 5.50MB 4.03% github.com/elastic/go-structform/gotype.getReflectFold
0 0% 90.74% 5.50MB 4.03% github.com/elastic/go-structform/gotype.getReflectFoldStruct
0 0% 90.74% 36.51MB 26.75% github.com/elastic/go-structform/gotype.getReflectFoldStruct.makeStructFold.func2
0 0% 90.74% 36.51MB 26.75% github.com/elastic/go-structform/gotype.getReflectFoldStruct.makeStructFold.makeFieldsFold.func3
0 0% 90.74% 25.01MB 18.32% github.com/elastic/go-structform/gotype.reFoldString
0 0% 90.74% 11.50MB 8.43% github.com/elastic/go-structform/gotype.reFoldUint64
0 0% 90.74% 5.11MB 3.74% github.com/spf13/cobra.(*Command).Execute
0 0% 90.74% 5.11MB 3.74% github.com/spf13/cobra.(*Command).ExecuteC
0 0% 90.74% 5.11MB 3.74% github.com/spf13/cobra.(*Command).execute
0 0% 90.74% 1.01MB 0.74% google.golang.org/protobuf/internal/filedesc.Builder.Build
0 0% 90.74% 1.01MB 0.74% google.golang.org/protobuf/internal/filedesc.newRawFile
0 0% 90.74% 1.53MB 1.12% google.golang.org/protobuf/internal/filetype.Builder.Build
0 0% 90.74% 3.50MB 2.57% k8s.io/apimachinery/pkg/runtime.(*Scheme).New
0 0% 90.74% 20.01MB 14.66% k8s.io/apimachinery/pkg/runtime.Decode (inline)
0 0% 90.74% 3.50MB 2.57% k8s.io/apimachinery/pkg/runtime.UseOrCreateObject
0 0% 90.74% 21.01MB 15.39% k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode
0 0% 90.74% 21.52MB 15.76% k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode
0 0% 90.74% 18.01MB 13.19% k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).unmarshal
0 0% 90.74% 16MB 11.72% k8s.io/apimachinery/pkg/util/wait.BackoffUntil
0 0% 90.74% 16MB 11.72% k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
0 0% 90.74% 16MB 11.72% k8s.io/apimachinery/pkg/util/wait.JitterUntil
0 0% 90.74% 16MB 11.72% k8s.io/apimachinery/pkg/util/wait.Until
0 0% 90.74% 19.51MB 14.29% k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive
0 0% 90.74% 1MB 0.73% k8s.io/client-go/kubernetes/typed/core/v1.(*pods).List
0 0% 90.74% 1MB 0.73% k8s.io/client-go/rest.Result.Into
0 0% 90.74% 20.01MB 14.66% k8s.io/client-go/rest/watch.(*Decoder).Decode
0 0% 90.74% 5.61MB 4.11% main.main
0 0% 90.74% 11.51MB 8.43% reflect.MakeSlice
0 0% 90.74% 15.60MB 11.42% runtime.doInit (inline)
0 0% 90.74% 15.60MB 11.42% runtime.doInit1
0 0% 90.74% 21.20MB 15.53% runtime.main
0 0% 90.74% 18.01MB 13.19% sigs.k8s.io/json.UnmarshalCaseSensitivePreserveInts (inline)
0 0% 90.74% 14.01MB 10.26% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).array
0 0% 90.74% 18.01MB 13.19% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).object
0 0% 90.74% 18.01MB 13.19% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).unmarshal
0 0% 90.74% 18.01MB 13.19% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).value
0 0% 90.74% 18.01MB 13.19% sigs.k8s.io/json/internal/golang/encoding/json.Unmarshal
0 0% 90.74% 1MB 0.73% sigs.k8s.io/json/internal/golang/encoding/json.indirect
0 0% 90.74% 2MB 1.47% strings.(*Builder).Grow
0 0% 90.74% 1.50MB 1.10% strings.Replace