Frequent restarts - kubernetes_metadata

[Raj_Etta]

Used the daemonset yaml at https://raw.githubusercontent.com/elastic/beats/master/deploy/kubernetes/filebeat-kubernetes.yaml with following added to filebeat-prospector(Version 6.2.4 and tried 6.3 also)

metadata:
  name: filebeat-prospectors
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  kubernetes.yml: |-
    - type: docker
      multiline.pattern: '^[[:digit:]]+'
      multiline.negate: true
      multiline.match: after
      exclude_lines: ['kafka']
      containers.ids:
      - "*"
      processors:
        - add_kubernetes_metadata:
            in_cluster: true

One of the error message from previously failed filebeat container. :

[signal SIGSEGV: segmentation violation code=0x1 addr=0x158 pc=0x13dd6fa]

goroutine 147 [running]:
github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.(*ContainerIndexer).GetIndexes(0xc420141d90, 0x0, 0xc420213230, 0xc420204120, 0xc420204060)
	/go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/indexers.go:161 +0x3a
github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.(*Indexers).GetIndexes(0xc420213230, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/indexers.go:76 +0x230
github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.(*kubernetesAnnotator).removePod(0xc4201541c0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/kubernetes.go:188 +0x3d
github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.(*kubernetesAnnotator).(github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.removePod)-fm(0x0)
	/go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/kubernetes.go:159 +0x34
github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.processEvent(0xc4203d6ed0, 0xc424673e90)
	/go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/kubernetes.go:175 +0x78
github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.(*kubernetesAnnotator).worker(0xc4201541c0)
	/go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/kubernetes.go:159 +0x26f
created by github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata.newKubernetesAnnotator
	/go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/kubernetes.go:113 +0x9f7

Sometimes, we also saw the filebeat container consuming high CPU and the log file only has "\\" repeated over and over.

[ruflin]

Any chance you could open a Github issue with this repo? Can you reproduce this reliably? If yes, please provide as many details about how to reproduce as possible in the Github issue. (yes, you already provided quite a bit of data, thanks for that).