Observed a panic: "invalid memory address or nil pointer dereference"

Elastic Stack Beats
1

Hello
Looks like I have the same issues as written in closed issue 14320: https://github.com/elastic/beats/issues/14320

So I am using: ( as directed in https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html )

curl -L -O https://raw.githubusercontent.com/elastic/beats/7.4/deploy/kubernetes/metricbeat-kubernetes.yaml

In that yaml it gives image: docker.elastic.co/beats/metricbeat:7.4.2

The first error I get:

E1104 18:35:51.712242       1 reflector.go:125] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: Failed to list *v1beta1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:kube-system:metricbeat" cannot list resource "replicasets" in API group "apps" at the cluster scope

I solved this with adding - replicasets in the clusterrole.

The second error:

E1104 18:35:51.800530       1 runtime.go:73] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 309 [running]:

Looks the same as in issue 14320.

I am using AWS EKS v1.13.

To complete it:

Here my commands:


MyCluster $ kubectl create -f metricbeat-kubernetes.yaml 

MyCluster $ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
(left out some lines )
kubernetes-dashboard-65c76f6c97-sddjx   1/1     Running   0          2d2h
metricbeat-54cfc89846-npx9d             0/1     Error     1          3s
metricbeat-fd29k                        1/1     Running   0          3s
metricbeat-grvhq                        1/1     Running   0          3s
metricbeat-n29zc                        1/1     Running   0          3s

MyCluster $ kubectl logs metricbeat-54cfc89846-npx9d
2019-11-04T18:40:21.193Z        INFO    instance/beat.go:607    Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2019-11-04T18:40:21.195Z        INFO    instance/beat.go:615    Beat ID: 633b1d8d-05f8-4923-bc6c-9774d8adc4c5
2019-11-04T18:40:21.196Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2019-11-04T18:40:21.196Z        INFO    [beat]  instance/beat.go:903    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "633b1d8d-05f8-4923-bc6c-9774d8adc4c5"}}}
2019-11-04T18:40:21.196Z        INFO    [beat]  instance/beat.go:912    Build info      {"system_info": {"build": {"commit": "15075156388b44390301f070960fd8aeac1c9712", "libbeat": "7.4.2", "time": "2019-10-28T19:49:40.000Z", "version": "7.4.2"}}}
2019-11-04T18:40:21.196Z        INFO    [beat]  instance/beat.go:915    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.9"}}}
2019-11-04T18:40:21.197Z        INFO    [beat]  instance/beat.go:919    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-11-02T19:24:48Z","containerized":false,"name":"ip-192-168-71-10.us-west-2.compute.internal","ip":["127.0.0.1/8","::1/128","192.168.71.10/19","fe80::44:55ff:fe11:1a7e/64","172.17.0.1/16","fe80::4088:62ff:feda:a13c/64","fe80::7806:18ff:fe2d:f6a0/64","192.168.88.134/19","fe80::c3:e5ff:fe58:5a18/64","fe80::908e:55ff:fe33:c313/64","fe80::e8e0:8ff:fea5:7184/64","fe80::4c75:a7ff:fe99:3da9/64"],"kernel_version":"4.14.138-114.102.amzn2.x86_64","mac":["02:44:55:11:1a:7e","02:42:0c:58:28:fc","42:88:62:da:a1:3c","7a:06:18:2d:f6:a0","02:c3:e5:58:5a:18","92:8e:55:33:c3:13","ea:e0:08:a5:71:84","4e:75:a7:99:3d:a9"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
(.... left out other lines )



2019-11-04T18:40:21.208Z        INFO    cfgfile/reload.go:171   Config reloader started
2019-11-04T18:40:21.217Z        INFO    cfgfile/reload.go:226   Loading of config files completed.
E1104 18:40:21.317827       1 runtime.go:73] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 295 [running]:
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x3fbb0e0, 0x6eebab0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:69 +0x7b
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 +0x82
panic(0x3fbb0e0, 0x6eebab0)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/elastic/beats/libbeat/common/kubernetes.(*MetaGeneratorConfig).ResourceMetadata(0xc0008875e0, 0x4a54f80, 0xc000ebdf10, 0x451844f)
        /go/src/github.com/elastic/beats/libbeat/common/kubernetes/metadata.go:113 +0x485
github.com/elastic/beats/metricbeat/module/kubernetes/util.NewResourceMetadataEnricher.func1(0xc00089af60, 0x4a54f80, 0xc000ebdf10)
        /go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/kubernetes.go:149 +0x1e9
github.com/elastic/beats/metricbeat/module/kubernetes/util.buildMetadataEnricher.func1(0x44e89a0, 0xc000ebdf10)
        /go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/kubernetes.go:268 +0xb1
github.com/elastic/beats/libbeat/common/kubernetes.ResourceEventHandlerFuncs.OnAdd(...)
        /go/src/github.com/elastic/beats/libbeat/common/kubernetes/eventhandler.go:54
github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).process.func1(0xc0003dfa40, 0x3d87fe0, 0xc000f13160, 0x0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:284 +0x1e0
github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).process(0xc0003dfa40, 0x4a92400, 0xc00087f440, 0x1)
        /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:292 +0x4d
github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).Start.func1()
        /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:247 +0x48
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0002e6280)
        /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0002e6280, 0x3b9aca00, 0x0, 0x1, 0xc0006eef00)

( left out other lines )
1 Like
2

I have discovered the cause , but not the reason.
In this Configmap it gives the metricsets that will be used by metricbeat

apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: monitoring
  labels:
    k8s-app: metricbeat
data:
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - state_node
        - state_deployment 
        - state_replicaset 
        - state_pod
        - state_container
        # Uncomment this to get k8s events:
        - event
      period: 10s
      host: ${NODE_NAME}
      hosts: ["kube-state-metrics:8080"]

Now the problem on my aws EKS cluster is that when I have this one enabled:

- state_deployment

then the metricbeat pod fails.
I have eks v1.13.

On my minikube it works fine.

So I disable - state_deployment and than the pod starts fine.

So why does it crash on - state_deployment ?

1 Like
3

Thanks for all the information! One quick question, what version of Metricbeat are you running? Seems like this bug was fixed several days ago so the upcoming release 7.5 version of Metricbeat will definitely have the fix in it.

4

I run docker.elastic.co/beats/metricbeat:7.4.2

5

Yeah I think this bug fix didn't get into 7.4.2. If you could give 7.5 a try when it releases, that should help. Thanks!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.