Can't run MetricBeat on Openshift

agiorgi · October 31, 2019, 4:13pm

Hi guys

I'm having issues while trying to setup metricbeat on our Openshift cluster.
Here are some details:
Elasticsearch Version: 7.4
MetricBeat version: 7.4
OpenShift & Kubernetes:

(OCP) Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.14-201908291507+c406751-dirty", GitCommit:"c406751", GitTreeState:"dirty", BuildDate:"2019-08-29T22:56:42Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

(KUBE) Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+520769a", GitCommit:"520769a", GitTreeState:"clean", BuildDate:"2019-10-11T01:55:01Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

I managed to get the DeamonSet to run fine, and ocp workers hosts metrics are getting indexed correctly in elasticsearch.
There are 3 metricbeat pods created, 2 of the deamonset, as i have 2 workers, and 1 for the deployments, which is failing to start (In the next two posts you'll find the trace of the error, i had to split it because of the 7000 chars per post limitation)

Pods are running in custom namespace named "metricbeat".
Kube-state-metric is already deployed in openshift in the namespace named "openshift-monitoring".

I applied the changes as per documentation instructions:
https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html#_red_hat_openshift_configuration
I just downloaded the provided manifest and customized as per instructions above, then deployed on openshift using "oc create -f manifest.yml"

I don't know what could be the cause of this error, and i can't find anything on google to help me out...
Let me know if need more details i can provide

Have you ever encountered this issue? Can you please help me out?

Many thanks!!!!
Alessandro

agiorgi · October 31, 2019, 4:14pm

The Deployments fails when starting metricbeat with the following error:

oc logs pod/metricbeat-7cb84b68d6-9ndr9

2019-10-31T15:53:10.521Z  INFO   instance/beat.go:607      Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]

2019-10-31T15:53:10.608Z  INFO   instance/beat.go:615      Beat ID: aca94366-da94-45eb-997d-935ff47c7db2

2019-10-31T15:53:10.610Z  INFO   [seccomp]    seccomp/seccomp.go:124    Syscall filter successfully installed

2019-10-31T15:53:10.610Z  INFO   [beat] instance/beat.go:903      Beat info    {"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "aca94366-da94-45eb-997d-935ff47c7db2"}}}

2019-10-31T15:53:10.610Z  INFO   [beat] instance/beat.go:912      Build info   {"system_info": {"build": {"commit": "12ee6cd05c1bfdc69721ddab1f473417b1514576", "libbeat": "7.4.1", "time": "2019-10-22T16:26:05.000Z", "version": "7.4.1"}}}

2019-10-31T15:53:10.610Z  INFO   [beat] instance/beat.go:915      Go runtime info       {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.9"}}}

2019-10-31T15:53:10.612Z  INFO   [beat] instance/beat.go:919      Host info    {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-10-22T17:34:29Z","containerized":false,"name":"metricbeat-7cb84b68d6-9ndr9","ip":["127.0.0.1/8","::1/128","10.131.1.196/23","fe80::24fe:afff:fe1e:8001/64"],"kernel_version":"4.18.0-80.11.2.el8_0.x86_64","mac":["0a:58:0a:83:01:c4"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}

2019-10-31T15:53:10.612Z  INFO   [beat] instance/beat.go:948      Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2019-10-31T15:53:09.330Z"}}}

2019-10-31T15:53:10.612Z  INFO   instance/beat.go:292      Setup Beat: metricbeat; Version: 7.4.1

2019-10-31T15:53:10.613Z  INFO   [index-management]  idxmgmt/std.go:178  Set output.elasticsearch.index to 'metricbeat-7.4.1' as ILM is enabled.

2019-10-31T15:53:10.613Z  INFO   elasticsearch/client.go:170      Elasticsearch url: http://itl-grsela-01:9200

2019-10-31T15:53:10.613Z  INFO   [publisher]  pipeline/module.go:97     Beat name: metricbeat-7cb84b68d6-9ndr9

2019-10-31T15:53:10.616Z  INFO   instance/beat.go:422      metricbeat start running.

2019-10-31T15:53:10.616Z  INFO   [monitoring] log/log.go:118      Starting metrics logging every 30s

2019-10-31T15:53:10.629Z  INFO   cfgfile/reload.go:171     Config reloader started

2019-10-31T15:53:10.643Z  INFO   cfgfile/reload.go:226     Loading of config files completed.

agiorgi · October 31, 2019, 4:14pm

E1031 15:53:10.747562       1 runtime.go:73] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)

goroutine 275 [running]:

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x3fbb0e0, 0x6eebab0)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:69 +0x7b

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 +0x82

panic(0x3fbb0e0, 0x6eebab0)

       /usr/local/go/src/runtime/panic.go:522 +0x1b5

github.com/elastic/beats/libbeat/common/kubernetes.(*MetaGeneratorConfig).ResourceMetadata(0xc00091aa00, 0x4a54f80, 0xc000d4a950, 0x451844f)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/metadata.go:113 +0x485

github.com/elastic/beats/metricbeat/module/kubernetes/util.NewResourceMetadataEnricher.func1(0xc0002a6fc0, 0x4a54f80, 0xc000d4a950)

       /go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/kubernetes.go:149 +0x1e9

github.com/elastic/beats/metricbeat/module/kubernetes/util.buildMetadataEnricher.func1(0x44e89a0, 0xc000d4a950)

       /go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/kubernetes.go:268 +0xb1

github.com/elastic/beats/libbeat/common/kubernetes.ResourceEventHandlerFuncs.OnAdd(...)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/eventhandler.go:54

github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).process.func1(0xc0001bf490, 0x3d87fe0, 0xc0012c84e0, 0x0, 0x0)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:284 +0x1e0

github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).process(0xc0001bf490, 0x4a92400, 0xc00030fc40, 0x1)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:292 +0x4d

github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).Start.func1()

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:247 +0x48

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0008f0000)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0008f0000, 0x3b9aca00, 0x0, 0x1, 0xc0003685a0)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc0008f0000, 0x3b9aca00, 0xc0003685a0)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d

created by github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).Start

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:246 +0x1ef

panic: runtime error: invalid memory address or nil pointer dereference [recovered]

       panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2c21015]

 

goroutine 275 [running]:

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105

panic(0x3fbb0e0, 0x6eebab0)

       /usr/local/go/src/runtime/panic.go:522 +0x1b5

github.com/elastic/beats/libbeat/common/kubernetes.(*MetaGeneratorConfig).ResourceMetadata(0xc00091aa00, 0x4a54f80, 0xc000d4a950, 0x451844f)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/metadata.go:113 +0x485

github.com/elastic/beats/metricbeat/module/kubernetes/util.NewResourceMetadataEnricher.func1(0xc0002a6fc0, 0x4a54f80, 0xc000d4a950)

       /go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/kubernetes.go:149 +0x1e9

github.com/elastic/beats/metricbeat/module/kubernetes/util.buildMetadataEnricher.func1(0x44e89a0, 0xc000d4a950)

       /go/src/github.com/elastic/beats/metricbeat/module/kubernetes/util/kubernetes.go:268 +0xb1

github.com/elastic/beats/libbeat/common/kubernetes.ResourceEventHandlerFuncs.OnAdd(...)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/eventhandler.go:54

github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).process.func1(0xc0001bf490, 0x3d87fe0, 0xc0012c84e0, 0x0, 0x0)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:284 +0x1e0

github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).process(0xc0001bf490, 0x4a92400, 0xc00030fc40, 0x1)

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:292 +0x4d

github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).Start.func1()

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:247 +0x48

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0008f0000)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0008f0000, 0x3b9aca00, 0x0, 0x1, 0xc0003685a0)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8

github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc0008f0000, 0x3b9aca00, 0xc0003685a0)

       /go/src/github.com/elastic/beats/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d

created by github.com/elastic/beats/libbeat/common/kubernetes.(*watcher).Start

       /go/src/github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:246 +0x1ef

Mario_Castro · November 1, 2019, 9:09pm

Hi @agiorgi

Just for completion, can you detail the version of Kubernetes, Kube State Metrics and Openshift you are using, please?

Thanks

agiorgi · November 5, 2019, 1:40pm

Hi @Mario_Castro,

Thanks for the quick reply and sorry for the delayed answer.
OCP version is 4.1.
Kube State Metrics version is 1.7.1 (we deployed a dedicated kube-state-metric pod inside metricbeat's namespace, instead of using the one we already had deployed in the openshift-monitoring namespace)
Kubernetes version is v1.13.4+285f2f5

We managed to get it working by downgrading MetricBeat to 7.3.2 (we were using 7.4.2).
Once downgraded to 7.3.2, metricbeat pods were not panicking anymore.
Turned out that even the DeamonSet was not working properly, as it was failing to send pod's metric because we had issues with the certificates (i could see the errors in the documents being indexed in ES), so the DeamonSet was only providing details for the hosts (OCP Workers) instead of the full metrics for the services running on top of kubernetes' pod.

Many thanks!
Alessandro

agiorgi · November 8, 2019, 10:49am

Hi,

Quick update:
I had a lapsus this morning about the issue i've reported. It might all have been due to incompatibility with the versions i was using.
By looking at the compatibility matrix of the kube-state-metric service, i can see that up to kube-state-metrics version 1.7.1, the related supported go-client version is v11.
I'm guessing that Metricbeat 7.4.2 has version 12 of the go-client embedded, that explains why i got a runtime-panic. i was running a newer and incompatible version of the client (v12) with an older version of the kube-state-metrics (v1.7.1).
Once i downgraded to Metricbeat 7.3.2 everything was working smoothly because, and i'm just guessing, 7.3.2 has embedded version 11 of the go client, which is compatible with kube-state-metrics 1.7.1.
Might that be the reason? Where can i find details about go-client version used by metricbeat?

Unfortunately i haven't tried this yet.
i'll update this thread as soon as i'll be able to verify this.

Cheers,
Alessandro

Mario_Castro · November 12, 2019, 12:15pm

Thank you for the update @agiorgi

system · December 10, 2019, 12:15pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.