# Can't run MetricBeat on Openshift

**URL:** <https://discuss.elastic.co/t/cant-run-metricbeat-on-openshift/206076>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [October 31, 2019, 4:13pm UTC](https://discuss.elastic.co/t/cant-run-metricbeat-on-openshift/206076 "2019-10-31T16:13:46Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![agiorgi](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@agiorgi](https://discuss.elastic.co/u/agiorgi)\
**Post date:** [October 31, 2019, 4:13pm UTC](https://discuss.elastic.co/t/cant-run-metricbeat-on-openshift/206076/1 "2019-10-31T16:13:46Z")

</div>

Hi guys 🙂

I'm having issues while trying to setup metricbeat on our Openshift cluster.  
Here are some details:  
Elasticsearch Version: 7.4  
MetricBeat version: 7.4  
OpenShift & Kubernetes:

```
(OCP) Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.14-201908291507+c406751-dirty", GitCommit:"c406751", GitTreeState:"dirty", BuildDate:"2019-08-29T22:56:42Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

(KUBE) Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+520769a", GitCommit:"520769a", GitTreeState:"clean", BuildDate:"2019-10-11T01:55:01Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

```

I managed to get the DeamonSet to run fine, and ocp workers hosts metrics are getting indexed correctly in elasticsearch.  
There are 3 metricbeat pods created, 2 of the deamonset, as i have 2 workers, and 1 for the deployments, which is failing to start (In the next two posts you'll find the trace of the error, i had to split it because of the 7000 chars per post limitation)

Pods are running in custom namespace named "metricbeat".  
Kube-state-metric is already deployed in openshift in the namespace named "openshift-monitoring".

I applied the changes as per documentation instructions:  
[https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html#\_red\_hat\_openshift\_configuration](https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html#_red_hat_openshift_configuration)  
I just downloaded the provided manifest and customized as per instructions above, then deployed on openshift using "oc create -f manifest.yml"

I don't know what could be the cause of this error, and i can't find anything on google to help me out...  
Let me know if need more details i can provide

Have you ever encountered this issue? Can you please help me out?

Many thanks!!!!  
Alessandro

---

<div class="post-metadata">

**Author:** ![agiorgi](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@agiorgi](https://discuss.elastic.co/u/agiorgi)\
**Post date:** [October 31, 2019, 4:14pm UTC](https://discuss.elastic.co/t/cant-run-metricbeat-on-openshift/206076/2 "2019-10-31T16:14:16Z")

</div>

The Deployments fails when starting metricbeat with the following error:

```
oc logs pod/metricbeat-7cb84b68d6-9ndr9

2019-10-31T15:53:10.521Z INFO instance/beat.go:607 Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]

2019-10-31T15:53:10.608Z INFO instance/beat.go:615 Beat ID: aca94366-da94-45eb-997d-935ff47c7db2

2019-10-31T15:53:10.610Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed

2019-10-31T15:53:10.610Z INFO [beat] instance/beat.go:903 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "aca94366-da94-45eb-997d-935ff47c7db2"}}}

2019-10-31T15:53:10.610Z INFO [beat] instance/beat.go:912 Build info {"system_info": {"build": {"commit": "12ee6cd05c1bfdc69721ddab1f473417b1514576", "libbeat": "7.4.1", "time": "2019-10-22T16:26:05.000Z", "version": "7.4.1"}}}

2019-10-31T15:53:10.610Z INFO [beat] instance/beat.go:915 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.9"}}}

2019-10-31T15:53:10.612Z INFO [beat] instance/beat.go:919 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-10-22T17:34:29Z","containerized":false,"name":"metricbeat-7cb84b68d6-9ndr9","ip":["127.0.0.1/8","::1/128","10.131.1.196/23","fe80::24fe:afff:fe1e:8001/64"],"kernel_version":"4.18.0-80.11.2.el8_0.x86_64","mac":["0a:58:0a:83:01:c4"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}

2019-10-31T15:53:10.612Z INFO [beat] instance/beat.go:948 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2019-10-31T15:53:09.330Z"}}}

2019-10-31T15:53:10.612Z INFO instance/beat.go:292 Setup Beat: metricbeat; Version: 7.4.1

2019-10-31T15:53:10.613Z INFO [index-management] idxmgmt/std.go:178 Set output.elasticsearch.index to 'metricbeat-7.4.1' as ILM is enabled.

2019-10-31T15:53:10.613Z INFO elasticsearch/client.go:170 Elasticsearch url: http://itl-grsela-01:9200

2019-10-31T15:53:10.613Z INFO [publisher] pipeline/module.go:97 Beat name: metricbeat-7cb84b68d6-9ndr9

2019-10-31T15:53:10.616Z INFO instance/beat.go:422 metricbeat start running.

2019-10-31T15:53:10.616Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s

2019-10-31T15:53:10.629Z INFO cfgfile/reload.go:171 Config reloader started

2019-10-31T15:53:10.643Z INFO cfgfile/reload.go:226 Loading of config files completed.
```

---

<div class="post-metadata">

**Author:** ![agiorgi](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@agiorgi](https://discuss.elastic.co/u/agiorgi)\
**Post date:** [October 31, 2019, 4:14pm UTC](https://discuss.elastic.co/t/cant-run-metricbeat-on-openshift/206076/3 "2019-10-31T16:14:23Z")

</div>

```
E1031 15:53:10.747562 1 runtime.go:73] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)

goroutine 275 [running]:

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x3fbb0e0, 0x6eebab0)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:69 +0x7b

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 +0x82

panic(0x3fbb0e0, 0x6eebab0)

       /usr/local/go/src/runtime/panic.go:522 +0x1b5

github.com/elastic/beats/libbeat/common/kubernetes.(*MetaGeneratorConfig).ResourceMetadata(0xc00091aa00, 0x4a54f80, 0xc000d4a950, 0x451844f)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/metadata.go:113 +0x485

github.com/elastic/beats/metricbeat/module/kubernetes/util.NewResourceMetadataEnricher.func1(0xc0002a6fc0, 0x4a54f80, 0xc000d4a950)

       /go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/kubernetes.go:149 +0x1e9

github.com/elastic/beats/metricbeat/module/kubernetes/util.buildMetadataEnricher.func1(0x44e89a0, 0xc000d4a950)

       /go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/kubernetes.go:268 +0xb1

github.com/elastic/beats/libbeat/common/kubernetes.ResourceEventHandlerFuncs.OnAdd(...)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/eventhandler.go:54

github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).process.func1(0xc0001bf490, 0x3d87fe0, 0xc0012c84e0, 0x0, 0x0)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:284 +0x1e0

github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).process(0xc0001bf490, 0x4a92400, 0xc00030fc40, 0x1)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:292 +0x4d

github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).Start.func1()

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:247 +0x48

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0008f0000)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0008f0000, 0x3b9aca00, 0x0, 0x1, 0xc0003685a0)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc0008f0000, 0x3b9aca00, 0xc0003685a0)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d

created by github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).Start

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:246 +0x1ef

panic: runtime error: invalid memory address or nil pointer dereference [recovered]

       panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2c21015]

 

goroutine 275 [running]:

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105

panic(0x3fbb0e0, 0x6eebab0)

       /usr/local/go/src/runtime/panic.go:522 +0x1b5

github.com/elastic/beats/libbeat/common/kubernetes.(*MetaGeneratorConfig).ResourceMetadata(0xc00091aa00, 0x4a54f80, 0xc000d4a950, 0x451844f)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/metadata.go:113 +0x485

github.com/elastic/beats/metricbeat/module/kubernetes/util.NewResourceMetadataEnricher.func1(0xc0002a6fc0, 0x4a54f80, 0xc000d4a950)

       /go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/kubernetes.go:149 +0x1e9

github.com/elastic/beats/metricbeat/module/kubernetes/util.buildMetadataEnricher.func1(0x44e89a0, 0xc000d4a950)

       /go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/kubernetes.go:268 +0xb1

github.com/elastic/beats/libbeat/common/kubernetes.ResourceEventHandlerFuncs.OnAdd(...)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/eventhandler.go:54

github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).process.func1(0xc0001bf490, 0x3d87fe0, 0xc0012c84e0, 0x0, 0x0)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:284 +0x1e0

github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).process(0xc0001bf490, 0x4a92400, 0xc00030fc40, 0x1)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:292 +0x4d

github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).Start.func1()

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:247 +0x48

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0008f0000)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0008f0000, 0x3b9aca00, 0x0, 0x1, 0xc0003685a0)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc0008f0000, 0x3b9aca00, 0xc0003685a0)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d

created by github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).Start

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:246 +0x1ef
```

---

<div class="post-metadata">

**Author:** ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)\
**Post date:** [November 1, 2019, 9:09pm UTC](https://discuss.elastic.co/t/cant-run-metricbeat-on-openshift/206076/4 "2019-11-01T21:09:13Z")

</div>

Hi @agiorgi 🙂

Just for completion, can you detail the version of Kubernetes, Kube State Metrics and Openshift you are using, please?

Thanks

---

<div class="post-metadata">

**Author:** ![agiorgi](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@agiorgi](https://discuss.elastic.co/u/agiorgi)\
**Post date:** [November 5, 2019, 1:40pm UTC](https://discuss.elastic.co/t/cant-run-metricbeat-on-openshift/206076/5 "2019-11-05T13:40:58Z")

</div>

Hi @Mario_Castro,

Thanks for the quick reply and sorry for the delayed answer.  
OCP version is 4.1.  
Kube State Metrics version is [1.7.1](http://quay.io/coreos/kube-state-metrics:v1.7.1) (we deployed a dedicated kube-state-metric pod inside metricbeat's namespace, instead of using the one we already had deployed in the openshift-monitoring namespace)  
Kubernetes version is v1.13.4+285f2f5

We managed to get it working by downgrading MetricBeat to 7.3.2 (we were using 7.4.2).  
Once downgraded to 7.3.2, metricbeat pods were not panicking anymore.  
Turned out that even the DeamonSet was not working properly, as it was failing to send pod's metric because we had issues with the certificates (i could see the errors in the documents being indexed in ES), so the DeamonSet was only providing details for the hosts (OCP Workers) instead of the full metrics for the services running on top of kubernetes' pod.

Many thanks!  
Alessandro

---

<div class="post-metadata">

**Author:** ![agiorgi](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@agiorgi](https://discuss.elastic.co/u/agiorgi)\
**Post date:** [November 8, 2019, 10:49am UTC](https://discuss.elastic.co/t/cant-run-metricbeat-on-openshift/206076/6 "2019-11-08T10:49:38Z")

</div>

Hi,

Quick update:  
I had a lapsus this morning about the issue i've reported. It might all have been due to incompatibility with the versions i was using.  
By looking at the [compatibility matrix](https://github.com/kubernetes/kube-state-metrics#compatibility-matrix) of the kube-state-metric service, i can see that up to kube-state-metrics version 1.7.1, the related supported go-client version is v11.  
I'm guessing that Metricbeat 7.4.2 has version 12 of the go-client embedded, that explains why i got a runtime-panic. i was running a newer and incompatible version of the client (v12) with an older version of the kube-state-metrics (v1.7.1).  
Once i downgraded to Metricbeat 7.3.2 everything was working smoothly because, and i'm just guessing, 7.3.2 has embedded version 11 of the go client, which is compatible with kube-state-metrics 1.7.1.  
Might that be the reason? Where can i find details about go-client version used by metricbeat?

Unfortunately i haven't tried this yet.  
i'll update this thread as soon as i'll be able to verify this.

Cheers,  
Alessandro

---

<div class="post-metadata">

**Author:** ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)\
**Post date:** [November 12, 2019, 12:15pm UTC](https://discuss.elastic.co/t/cant-run-metricbeat-on-openshift/206076/7 "2019-11-12T12:15:37Z")

</div>

Thank you for the update @agiorgi 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 10, 2019, 12:15pm UTC](https://discuss.elastic.co/t/cant-run-metricbeat-on-openshift/206076/8 "2019-12-10T12:15:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.