Unable to start Metricbeat in OpenShift

ralfeus · April 18, 2018, 12:19pm

Hi.
I'm trying to run Metricbeat in OpenShift:

oc v3.9.0-alpha.3+78ddc10
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://openshift-ph.lan.net:8443
kubernetes v1.6.1+5115d708d7

I've installed with oc create -f metricbeat-kubernetes.yaml, which created all resources. I've just modified it so namespace "logging" is used instead of "kube-system".
There are 6 metricbeat-* pods created:

one is probably created by deployment and is running fine.
five are created by daemonset (I have 3 masters and 2 nodes) and those ones fail.
Logs show:

Exiting: error loading config file: open /etc/metricbeat.yml: permission denied

I read about possible issues of using subPath in configmap. So I tried to remove subPath from the daemonset definition so it looked like this (only relevant parts are specified):

spec:
  template:
    spec:
      containers:
      - args:
        - -c
        - /etc/metricbeat/metricbeat.yml
        volumeMounts:
        - mountPath: /etc/metricbeat
          name: config
          readOnly: true

That gave same error:

Exiting: error loading config file: open /etc/metricbeat/metricbeat.yml: permission denied

What else can I check?

exekias · April 18, 2018, 3:13pm

Hi @ralfeus,

Could you share your full metricbeat-kubernetes.yml?
It's weird that permissions are working for the Deployment but not for the ones spawn by the DaemonSet.

Best regards

ralfeus · April 18, 2018, 7:29pm

Here it is:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-config
  namespace: logging
  labels:
    k8s-app: metricbeat
    kubernetes.io/cluster-service: "true"
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false

    processors:
      - add_cloud_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:logging-es}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-modules
  namespace: logging
  labels:
    k8s-app: metricbeat
    kubernetes.io/cluster-service: "true"
data:
  system.yml: |-
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        #- core
        #- diskio
        #- socket
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5      # include top 5 processes by CPU
        by_memory: 5   # include top 5 processes by memory

    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      period: 10s
      hosts: ["localhost:10255"]
---
# Deploy a Metricbeat instance per node for node metrics retrieval
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: metricbeat
  namespace: logging
  labels:
    k8s-app: metricbeat
    kubernetes.io/cluster-service: "true"
    component: metricbeat
spec:
  template:
    metadata:
      labels:
        k8s-app: metricbeat
        kubernetes.io/cluster-service: "true"
        component: metricbeat
    spec:
      serviceAccountName: metricbeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      containers:
      - name: metricbeat
        image: docker.elastic.co/beats/metricbeat:6.2.3
        args: [
          "-c", "/etc/metricbeat.yml",
          "-e",
          "-system.hostfs=/hostfs",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: "logging-es"
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: fluentd
        - name: ELASTICSEARCH_PASSWORD
          value: changeme
        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        securityContext:
          privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/metricbeat.yml
          readOnly: true
          subPath: metricbeat.yml
        - name: modules
          mountPath: /usr/share/metricbeat/modules.d
          readOnly: true
        - name: dockersock
          mountPath: /var/run/docker.sock
        - name: proc
          mountPath: /hostfs/proc
          readOnly: true
        - name: cgroup
          mountPath: /hostfs/sys/fs/cgroup
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: cgroup
        hostPath:
          path: /sys/fs/cgroup
      - name: dockersock
        hostPath:
          path: /var/run/docker.sock
      - name: config
        configMap:
          defaultMode: 0600
          name: metricbeat-config
      - name: modules
        configMap:
          defaultMode: 0600
          name: metricbeat-daemonset-modules
      - name: data
        emptyDir: {}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: logging
  labels:
    k8s-app: metricbeat
    kubernetes.io/cluster-service: "true"
data:
  # This module requires `kube-state-metrics` up and running under `logging` namespace
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_pod
        - state_container
        # Uncomment this to get k8s events:
        #- event
      period: 10s
      hosts: ["kube-state-metrics:8080"]

ralfeus · April 18, 2018, 7:30pm

Second part (didn't fit to 7000 symbols)

---
# Deploy singleton instance in the whole cluster for some unique data sources, like kube-state-metrics
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: metricbeat
  namespace: logging
  labels:
    k8s-app: metricbeat
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: metricbeat
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: metricbeat
        image: docker.elastic.co/beats/metricbeat:6.2.3
        args: [
          "-c", "/etc/metricbeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: logging-es
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: fluentd
        - name: ELASTICSEARCH_PASSWORD
          value: changeme
        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/metricbeat.yml
          readOnly: true
          subPath: metricbeat.yml
        - name: modules
          mountPath: /usr/share/metricbeat/modules.d
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: metricbeat-config
      - name: modules
        configMap:
          defaultMode: 0600
          name: metricbeat-deployment-modules
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: metricbeat
subjects:
- kind: ServiceAccount
  name: metricbeat
  namespace: logging
roleRef:
  kind: ClusterRole
  name: metricbeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: metricbeat
  labels:
    k8s-app: metricbeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - events
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metricbeat
  namespace: logging
  labels:
    k8s-app: metricbeat
---

exekias · April 19, 2018, 9:41am

I see you added this:

But my guess is that you still need the runAsuser: 0 here, to ensure that Metricbeat runs as root and can fetch the config. Could you give that a try?

Also, what's the reason to enable privileged mode?

Best regards

ralfeus · April 19, 2018, 9:56am

Yeah, that helped!
Thanks a lot

The reason was that at the very beginning pods could not start with error that anyuid can't use host network, which was true as anyuid scc doesn't allow host networks. Since we had already installled fluentd I checked that it uses privileged scc so did same for Metricbeat.

Thanks again.

exekias · April 19, 2018, 10:31am

Nice, if I understand it correctly, that's the only change you needed to have it working? We can push that to our repo to ease the process for Openshift users. Definitely, this is a use case we want to support

system · May 17, 2018, 10:31am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Can't run MetricBeat on Openshift Beats metricbeat	7	776	December 10, 2019
HTTP error 403 in pod - Metricbeat in Openshift Beats metricbeat	3	1351	August 17, 2020
MetricBeat on Openshift Beats metricbeat	2	307	August 25, 2020
Meetricbeat 7.6.2 on Openshift 4.3 Beats metricbeat	1	335	May 28, 2020
Use Metricbeat per Openshift namespace (not per cluster) Beats metricbeat	1	317	March 5, 2019

Unable to start Metricbeat in OpenShift

Related topics