ralfeus
(Mychajlo Chodorev)
April 18, 2018, 12:19pm
1
Hi.
I'm trying to run Metricbeat in OpenShift:
oc v3.9.0-alpha.3+78ddc10
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO
Server https://openshift-ph.lan.net:8443
kubernetes v1.6.1+5115d708d7
I've installed with oc create -f metricbeat-kubernetes.yaml, which created all resources. I've just modified it so namespace "logging" is used instead of "kube-system".
There are 6 metricbeat-* pods created:
one is probably created by deployment and is running fine.
five are created by daemonset (I have 3 masters and 2 nodes) and those ones fail.
Logs show:
Exiting: error loading config file: open /etc/metricbeat.yml: permission denied
I read about possible issues of using subPath in configmap. So I tried to remove subPath from the daemonset definition so it looked like this (only relevant parts are specified):
spec:
template:
spec:
containers:
- args:
- -c
- /etc/metricbeat/metricbeat.yml
volumeMounts:
- mountPath: /etc/metricbeat
name: config
readOnly: true
That gave same error:
Exiting: error loading config file: open /etc/metricbeat/metricbeat.yml: permission denied
What else can I check?
exekias
(Carlos Pérez Aradros)
April 18, 2018, 3:13pm
2
Hi @ralfeus ,
Could you share your full metricbeat-kubernetes.yml
?
It's weird that permissions are working for the Deployment but not for the ones spawn by the DaemonSet.
Best regards
ralfeus
(Mychajlo Chodorev)
April 18, 2018, 7:29pm
3
Here it is:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-config
namespace: logging
labels:
k8s-app: metricbeat
kubernetes.io/cluster-service: "true"
data:
metricbeat.yml: |-
metricbeat.config.modules:
# Mounted `metricbeat-daemonset-modules` configmap:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
processors:
- add_cloud_metadata:
cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:logging-es}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-daemonset-modules
namespace: logging
labels:
k8s-app: metricbeat
kubernetes.io/cluster-service: "true"
data:
system.yml: |-
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
#- core
#- diskio
#- socket
processes: ['.*']
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
kubernetes.yml: |-
- module: kubernetes
metricsets:
- node
- system
- pod
- container
- volume
period: 10s
hosts: ["localhost:10255"]
---
# Deploy a Metricbeat instance per node for node metrics retrieval
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: metricbeat
namespace: logging
labels:
k8s-app: metricbeat
kubernetes.io/cluster-service: "true"
component: metricbeat
spec:
template:
metadata:
labels:
k8s-app: metricbeat
kubernetes.io/cluster-service: "true"
component: metricbeat
spec:
serviceAccountName: metricbeat
terminationGracePeriodSeconds: 30
hostNetwork: true
containers:
- name: metricbeat
image: docker.elastic.co/beats/metricbeat:6.2.3
args: [
"-c", "/etc/metricbeat.yml",
"-e",
"-system.hostfs=/hostfs",
]
env:
- name: ELASTICSEARCH_HOST
value: "logging-es"
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: fluentd
- name: ELASTICSEARCH_PASSWORD
value: changeme
- name: ELASTIC_CLOUD_ID
value:
- name: ELASTIC_CLOUD_AUTH
value:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
securityContext:
privileged: true
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/metricbeat.yml
readOnly: true
subPath: metricbeat.yml
- name: modules
mountPath: /usr/share/metricbeat/modules.d
readOnly: true
- name: dockersock
mountPath: /var/run/docker.sock
- name: proc
mountPath: /hostfs/proc
readOnly: true
- name: cgroup
mountPath: /hostfs/sys/fs/cgroup
readOnly: true
volumes:
- name: proc
hostPath:
path: /proc
- name: cgroup
hostPath:
path: /sys/fs/cgroup
- name: dockersock
hostPath:
path: /var/run/docker.sock
- name: config
configMap:
defaultMode: 0600
name: metricbeat-config
- name: modules
configMap:
defaultMode: 0600
name: metricbeat-daemonset-modules
- name: data
emptyDir: {}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-deployment-modules
namespace: logging
labels:
k8s-app: metricbeat
kubernetes.io/cluster-service: "true"
data:
# This module requires `kube-state-metrics` up and running under `logging` namespace
kubernetes.yml: |-
- module: kubernetes
metricsets:
- state_node
- state_deployment
- state_replicaset
- state_pod
- state_container
# Uncomment this to get k8s events:
#- event
period: 10s
hosts: ["kube-state-metrics:8080"]
ralfeus
(Mychajlo Chodorev)
April 18, 2018, 7:30pm
4
Second part (didn't fit to 7000 symbols)
---
# Deploy singleton instance in the whole cluster for some unique data sources, like kube-state-metrics
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: metricbeat
namespace: logging
labels:
k8s-app: metricbeat
kubernetes.io/cluster-service: "true"
spec:
template:
metadata:
labels:
k8s-app: metricbeat
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: metricbeat
image: docker.elastic.co/beats/metricbeat:6.2.3
args: [
"-c", "/etc/metricbeat.yml",
"-e",
]
env:
- name: ELASTICSEARCH_HOST
value: logging-es
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: fluentd
- name: ELASTICSEARCH_PASSWORD
value: changeme
- name: ELASTIC_CLOUD_ID
value:
- name: ELASTIC_CLOUD_AUTH
value:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/metricbeat.yml
readOnly: true
subPath: metricbeat.yml
- name: modules
mountPath: /usr/share/metricbeat/modules.d
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: metricbeat-config
- name: modules
configMap:
defaultMode: 0600
name: metricbeat-deployment-modules
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: metricbeat
subjects:
- kind: ServiceAccount
name: metricbeat
namespace: logging
roleRef:
kind: ClusterRole
name: metricbeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: metricbeat
labels:
k8s-app: metricbeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- events
- pods
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metricbeat
namespace: logging
labels:
k8s-app: metricbeat
---
exekias
(Carlos Pérez Aradros)
April 19, 2018, 9:41am
5
I see you added this:
ralfeus:
privileged: true
But my guess is that you still need the runAsuser: 0
here, to ensure that Metricbeat runs as root and can fetch the config. Could you give that a try?
Also, what's the reason to enable privileged mode?
Best regards
ralfeus
(Mychajlo Chodorev)
April 19, 2018, 9:56am
6
Yeah, that helped!
Thanks a lot
The reason was that at the very beginning pods could not start with error that anyuid can't use host network, which was true as anyuid scc doesn't allow host networks. Since we had already installled fluentd I checked that it uses privileged scc so did same for Metricbeat.
Thanks again.
exekias
(Carlos Pérez Aradros)
April 19, 2018, 10:31am
7
Nice, if I understand it correctly, that's the only change you needed to have it working? We can push that to our repo to ease the process for Openshift users. Definitely, this is a use case we want to support
1 Like
system
(system)
Closed
May 17, 2018, 10:31am
8
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.