ethrbunny
October 16, 2018, 2:00pm
1
Ive created a filebeat daemonset for my k8 cluster using (roughly) these params: https://github.com/elastic/beats/blob/master/deploy/kubernetes/filebeat/filebeat-daemonset.yaml .
Of 5 cluster hosts 4 are working fine and Im getting data. On the 5th Im seeing this error message:
2018-10-16T13:53:55.243Z INFO instance/beat.go:544 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2018-10-16T13:53:55.244Z INFO instance/beat.go:315 filebeat stopped.
2018-10-16T13:53:55.244Z ERROR instance/beat.go:743 Exiting: Failed to create Beat meta file: open /usr/share/filebeat/data/meta.json.new: permission denied
Exiting: Failed to create Beat meta file: open /usr/share/filebeat/data/meta.json.new: permission denied
The 5 hosts should be identical. (They were provisioned via puppet).
When I look for that path (/usr/share/filebeat/data) I don't see it on any of the 5 hosts. For some reason filebeat doesn't seem to complain on 4/5 though.
Any ideas?
steffens
October 16, 2018, 9:33pm
2
Filebeat needs to be the owner of /usr/share/filebeat. Are you sure the directory does not exist? The logs indicate filebeat even sets the home path to /usr/share/filebeat.
How does your setup differ from the docker images + daemonset in the beats repo?
ethrbunny
October 17, 2018, 10:23am
3
/usr/share/filebeat is root:root on all 5 machines. The 'data' folder doesn't exist on any of them.
steffens
October 17, 2018, 11:44am
4
How does your setup differ from the docker images + daemonset in the beats repo?
ethrbunny
October 17, 2018, 10:50pm
5
I pointed it at a logstash host. I can't think of anything else major (or minor for that matter).
exekias
October 19, 2018, 8:50am
6
Hi @ethrbunny , could you please report the manifest you are using and Kubernetes version?
Our default manifest is supposed to create the data folder for you, with the correct permissions.
Best regards
ethrbunny
October 19, 2018, 9:57am
7
Kubernetes 1.12
Here's the manifest:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.config:
inputs:
# Mounted `filebeat-inputs` configmap:
path: ${path.config}/inputs.d/*.yml
# Reload inputs configs as they change:
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
# To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this:
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
templates:
# - condition:
# regexp:
# kubernetes.labels.yourlabel: '.*'
config:
- type: docker
containers.ids:
- "${data.kubernetes.container.id}"
# Point 2, add custom fields to events:
fields:
yourdesiredfield: "${data.kubernentes.labels.yourlabel}"
processors:
- add_cloud_metadata:
output.logstash:
hosts: ["10.95.96.75:5044"]
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-inputs
namespace: kube-system
labels:
k8s-app: filebeat
data:
kubernetes.yml: |-
- type: docker
containers.ids:
- "*"
processors:
- add_kubernetes_metadata:
in_cluster: true
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
spec:
template:
metadata:
labels:
k8s-app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:6.4.1
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
# - name: ELASTICSEARCH_HOST
# value: elasticsearch
# - name: ELASTICSEARCH_PORT
# value: "9200"
# - name: ELASTICSEARCH_USERNAME
# value: elastic
# - name: ELASTICSEARCH_PASSWORD
# value: changeme
# - name: ELASTIC_CLOUD_ID
# value:
# - name: ELASTIC_CLOUD_AUTH
# value:
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: inputs
mountPath: /usr/share/filebeat/inputs.d
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: msgs
mountPath: /var/log/messages
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: msgs
hostPath:
path: /var/log/messages
- name: inputs
configMap:
defaultMode: 0600
name: filebeat-inputs
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: kube-system
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
---
system
November 16, 2018, 9:57am
8
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.