# Observed a panic: "invalid memory address or nil pointer dereference"
**URL:** https://discuss.elastic.co/t/observed-a-panic-invalid-memory-address-or-nil-pointer-dereference/206465
**Category:** Beats
**Tags:** metricbeat
**Created:** [November 4, 2019, 7:23pm UTC](https://discuss.elastic.co/t/observed-a-panic-invalid-memory-address-or-nil-pointer-dereference/206465 "2019-11-04T19:23:26Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@1robroos](https://discuss.elastic.co/u/1robroos)
#### Post date: [November 4, 2019, 7:23pm UTC](https://discuss.elastic.co/t/observed-a-panic-invalid-memory-address-or-nil-pointer-dereference/206465/1 "2019-11-04T19:23:27Z")
Hello
Looks like I have the same issues as written in closed issue 14320: [https://github.com/elastic/beats/issues/14320](https://github.com/elastic/beats/issues/14320)
So I am using: ( as directed in [https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html](https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html) )
```auto
curl -L -O https://raw.githubusercontent.com/elastic/beats/7.4/deploy/kubernetes/metricbeat-kubernetes.yaml
```
In that yaml it gives `image: docker.elastic.co/beats/metricbeat:7.4.2`
The first error I get:
```auto
E1104 18:35:51.712242 1 reflector.go:125] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: Failed to list *v1beta1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:kube-system:metricbeat" cannot list resource "replicasets" in API group "apps" at the cluster scope
```
I solved this with adding ` - replicasets` in the clusterrole.
The second error:
```auto
E1104 18:35:51.800530 1 runtime.go:73] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 309 [running]:
```
Looks the same as in issue 14320.
I am using AWS EKS v1.13.
To complete it:
Here my commands:
```auto
MyCluster $ kubectl create -f metricbeat-kubernetes.yaml
MyCluster $ kubectl get pods
NAME READY STATUS RESTARTS AGE
(left out some lines )
kubernetes-dashboard-65c76f6c97-sddjx 1/1 Running 0 2d2h
metricbeat-54cfc89846-npx9d 0/1 Error 1 3s
metricbeat-fd29k 1/1 Running 0 3s
metricbeat-grvhq 1/1 Running 0 3s
metricbeat-n29zc 1/1 Running 0 3s
MyCluster $ kubectl logs metricbeat-54cfc89846-npx9d
2019-11-04T18:40:21.193Z INFO instance/beat.go:607 Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2019-11-04T18:40:21.195Z INFO instance/beat.go:615 Beat ID: 633b1d8d-05f8-4923-bc6c-9774d8adc4c5
2019-11-04T18:40:21.196Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2019-11-04T18:40:21.196Z INFO [beat] instance/beat.go:903 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "633b1d8d-05f8-4923-bc6c-9774d8adc4c5"}}}
2019-11-04T18:40:21.196Z INFO [beat] instance/beat.go:912 Build info {"system_info": {"build": {"commit": "15075156388b44390301f070960fd8aeac1c9712", "libbeat": "7.4.2", "time": "2019-10-28T19:49:40.000Z", "version": "7.4.2"}}}
2019-11-04T18:40:21.196Z INFO [beat] instance/beat.go:915 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.9"}}}
2019-11-04T18:40:21.197Z INFO [beat] instance/beat.go:919 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-11-02T19:24:48Z","containerized":false,"name":"ip-192-168-71-10.us-west-2.compute.internal","ip":["127.0.0.1/8","::1/128","192.168.71.10/19","fe80::44:55ff:fe11:1a7e/64","172.17.0.1/16","fe80::4088:62ff:feda:a13c/64","fe80::7806:18ff:fe2d:f6a0/64","192.168.88.134/19","fe80::c3:e5ff:fe58:5a18/64","fe80::908e:55ff:fe33:c313/64","fe80::e8e0:8ff:fea5:7184/64","fe80::4c75:a7ff:fe99:3da9/64"],"kernel_version":"4.14.138-114.102.amzn2.x86_64","mac":["02:44:55:11:1a:7e","02:42:0c:58:28:fc","42:88:62:da:a1:3c","7a:06:18:2d:f6:a0","02:c3:e5:58:5a:18","92:8e:55:33:c3:13","ea:e0:08:a5:71:84","4e:75:a7:99:3d:a9"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
(.... left out other lines )
2019-11-04T18:40:21.208Z INFO cfgfile/reload.go:171 Config reloader started
2019-11-04T18:40:21.217Z INFO cfgfile/reload.go:226 Loading of config files completed.
E1104 18:40:21.317827 1 runtime.go:73] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 295 [running]:
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x3fbb0e0, 0x6eebab0)
/go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:69 +0x7b
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 +0x82
panic(0x3fbb0e0, 0x6eebab0)
/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/elastic/beats/libbeat/common/kubernetes.(*MetaGeneratorConfig).ResourceMetadata(0xc0008875e0, 0x4a54f80, 0xc000ebdf10, 0x451844f)
/go/src/github.com/elastic/beats/libbeat/common/kubernetes/metadata.go:113 +0x485
github.com/elastic/beats/metricbeat/module/kubernetes/util.NewResourceMetadataEnricher.func1(0xc00089af60, 0x4a54f80, 0xc000ebdf10)
/go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/kubernetes.go:149 +0x1e9
github.com/elastic/beats/metricbeat/module/kubernetes/util.buildMetadataEnricher.func1(0x44e89a0, 0xc000ebdf10)
/go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/kubernetes.go:268 +0xb1
github.com/elastic/beats/libbeat/common/kubernetes.ResourceEventHandlerFuncs.OnAdd(...)
/go/src/github.com/elastic/beats/libbeat/common/kubernetes/eventhandler.go:54
github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).process.func1(0xc0003dfa40, 0x3d87fe0, 0xc000f13160, 0x0, 0x0)
/go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:284 +0x1e0
github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).process(0xc0003dfa40, 0x4a92400, 0xc00087f440, 0x1)
/go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:292 +0x4d
github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).Start.func1()
/go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:247 +0x48
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0002e6280)
/go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54
github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0002e6280, 0x3b9aca00, 0x0, 0x1, 0xc0006eef00)
( left out other lines )
```
---
### Author:  [@1robroos](https://discuss.elastic.co/u/1robroos)
#### Post date: [November 14, 2019, 3:41pm UTC](https://discuss.elastic.co/t/observed-a-panic-invalid-memory-address-or-nil-pointer-dereference/206465/2 "2019-11-14T15:41:52Z")
I have discovered the cause , but not the reason.
In this Configmap it gives the metricsets that will be used by metricbeat
```auto
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-deployment-modules
namespace: monitoring
labels:
k8s-app: metricbeat
data:
kubernetes.yml: |-
- module: kubernetes
metricsets:
- state_node
- state_deployment
- state_replicaset
- state_pod
- state_container
# Uncomment this to get k8s events:
- event
period: 10s
host: ${NODE_NAME}
hosts: ["kube-state-metrics:8080"]
```
Now the problem on my aws EKS cluster is that when I have this one enabled:
```auto
- state_deployment
```
then the metricbeat pod fails.
I have eks v1.13.
On my minikube it works fine.
So I disable `- state_deployment ` and than the pod starts fine.
So why does it crash on `- state_deployment ` ?
---
### Author:  [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)
#### Post date: [November 15, 2019, 3:38pm UTC](https://discuss.elastic.co/t/observed-a-panic-invalid-memory-address-or-nil-pointer-dereference/206465/3 "2019-11-15T15:38:58Z")
Thanks for all the information! One quick question, what version of Metricbeat are you running? Seems like this bug was fixed several days ago so the upcoming release 7.5 version of Metricbeat will definitely have the fix in it.
---
### Author:  [@1robroos](https://discuss.elastic.co/u/1robroos)
#### Post date: [November 15, 2019, 9:17pm UTC](https://discuss.elastic.co/t/observed-a-panic-invalid-memory-address-or-nil-pointer-dereference/206465/4 "2019-11-15T21:17:11Z")
I run [docker.elastic.co/beats/metricbeat:7.4.2](http://docker.elastic.co/beats/metricbeat:7.4.2)
---
### Author:  [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)
#### Post date: [November 16, 2019, 4:20pm UTC](https://discuss.elastic.co/t/observed-a-panic-invalid-memory-address-or-nil-pointer-dereference/206465/5 "2019-11-16T16:20:00Z")
Yeah I think this bug fix didn't get into 7.4.2. If you could give 7.5 a try when it releases, that should help. Thanks!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 14, 2019, 4:20pm UTC](https://discuss.elastic.co/t/observed-a-panic-invalid-memory-address-or-nil-pointer-dereference/206465/6 "2019-12-14T16:20:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.