Hi folks,
Apologies if this has already been mentioned, I couldn't find much reference to it.
Would love to get some input as to what the cause of the error might be, as I've not managed to figure it out. I can use filebeat's processors to remove the message as it's causing a bit of noise in our logs, but would rather get to the root of the problem.
I've seen this with Filebeat 6.4.0, 6.3.x and 6.2.4 using the elastic docker images on AKS. I've not yet tried other managed kubernetes services like AWS EKS or Google's GKE, but will update this as I find things out.
I have filebeat running on AKS, sending logs to AWS' ElasticSearch service. Large parts of the kuvernetes manifests have been borrowed from the elastic repo. We are occasionally receiving error messages about attempting to enrich log messages with kubernetes metadata:
kubernetes: Watching API error unexpected EOF.
Any help would be greatly appreciated. Cheers,
Alexander
Here is the filebeat config:
logging.json: true
filebeat.registry_file: /var/lib/filebeat/registry/registry
filebeat.config:
modules: []
filebeat.inputs:
- type: docker
containers.ids:
- "*"
processors:
- add_kubernetes_metadata:
in_cluster: true
- drop_event:
when:
or:
- equals:
kubernetes.container.name: tunnel-front
- equals:
kubernetes.container.name: azureproxy
- equals:
kubernetes.container.name: redirector
- contains:
kubernetes.container.name: filebeat
- equals:
kubernetes.container.name: kube-proxy
- equals:
kubernetes.container.name: kube-svc-redirect
- equals:
kubernetes.container.name: kube-dns
- equals:
kubernetes.container.name: heapster
- equals:
kubernetes.container.name: main
- equals:
kubernetes.container.name: kubernetes-dashboard
json.keys_under_root: true
json.add_error_key: true
json.ignore_decoding_error: true
output.elasticsearch:
hosts: ['<redacted>']
protocol: "https"
Here is the kubernetes manifest:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: default
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
data:
filebeat.yml: |-
logging.json: true
filebeat.registry_file: /var/lib/filebeat/registry/registry
filebeat.config:
modules: []
filebeat.inputs:
- type: docker
containers.ids:
- "*"
processors:
- add_kubernetes_metadata:
in_cluster: true
- drop_event:
when:
or:
- equals:
kubernetes.container.name: tunnel-front
- equals:
kubernetes.container.name: azureproxy
- equals:
kubernetes.container.name: redirector
- contains:
kubernetes.container.name: filebeat
- equals:
kubernetes.container.name: kube-proxy
- equals:
kubernetes.container.name: kube-svc-redirect
- equals:
kubernetes.container.name: kube-dns
- equals:
kubernetes.container.name: heapster
- equals:
kubernetes.container.name: main
- equals:
kubernetes.container.name: kubernetes-dashboard
json.keys_under_root: true
json.add_error_key: true
json.ignore_decoding_error: true
output.elasticsearch:
hosts: ['<redacted>']
protocol: "https"
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: filebeat
namespace: default
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
spec:
template:
metadata:
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:6.4.0
args: [
"-c", "/etc/filebeat.yml",
"-e"
]
securityContext:
runAsUser: 0
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
- name: filebeat-registry
mountPath: /var/lib/filebeat/registry
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: varlog
hostPath:
path: /var/log
- name: data
hostPath:
type: DirectoryOrCreate
path: /usr/share/filebeat/data
- name: filebeat-registry
hostPath:
type: DirectoryOrCreate
path: /var/lib/filebeat/registry
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: default
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: default
labels:
k8s-app: filebeat
---