So ive mounted ELK stack with filebeat in a kubernetes enviroment, im parsing all the logs correctly, only problem is the kibana json-logs format that get error
failed to format message from /var/lib/docker/containers/b685d94ec5e83c08cbe7728bcc9ebc3827cf2015c25490fb8d62e5c16c12b8ba/b685d94ec5e83c08cbe7728bcc9ebc3827cf2015c25490fb8d62e5c16c12b8ba-json.log
So did a kubectl describe pods and realized that docker container was kibana. Version 6.5.2
Filebeat configuration:
--- apiVersion: v1 kind: ConfigMap metadata: name: filebeat-config namespace: kube-system labels: k8s-app: filebeat data: filebeat.yml: |- filebeat.config: inputs: path: ${path.config}/inputs.d/*.yml reload.enabled: false modules: path: ${path.config}/modules.d/*.yml reload.enabled: false processors: - add_cloud_metadata: - drop_fields: when: has_fields: ['kubernetes.labels.app'] fields: - 'kubernetes.labels.app' output.elasticsearch: hosts: ['http://elasticsearch.whitenfv.svc.cluster.local:9200'] --- apiVersion: v1 kind: ConfigMap metadata: name: filebeat-inputs namespace: kube-system labels: k8s-app: filebeat data: kubernetes.yml: |- - type: docker json.keys_under_root: false json.add_error_key: false json.ignore_decoding_error: true containers.ids: - "*" processors: - add_kubernetes_metadata: in_cluster: true --- apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: filebeat namespace: kube-system labels: k8s-app: filebeat spec: template: metadata: labels: k8s-app: filebeat spec: serviceAccountName: filebeat terminationGracePeriodSeconds: 30 containers: - name: filebeat image: {{ filebeat_image_full }} args: [ "-c", "/etc/filebeat.yml", "-e", ] securityContext: runAsUser: 0 resources: limits: memory: 200Mi requests: cpu: 100m memory: 100Mi volumeMounts: - name: config mountPath: /etc/filebeat.yml readOnly: true subPath: filebeat.yml - name: inputs mountPath: /usr/share/filebeat/inputs.d readOnly: true - name: data mountPath: /usr/share/filebeat/data - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true volumes: - name: config configMap: defaultMode: 0600 name: filebeat-config - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: inputs configMap: defaultMode: 0600 name: filebeat-inputs - name: data hostPath: path: /var/lib/filebeat-data type: DirectoryOrCreate --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: filebeat subjects: - kind: ServiceAccount name: filebeat namespace: kube-system roleRef: kind: ClusterRole name: filebeat apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: filebeat labels: k8s-app: filebeat rules: - apiGroups: [""] resources: - namespaces - pods verbs: - get - watch - list --- apiVersion: v1 kind: ServiceAccount metadata: name: filebeat namespace: kube-system labels: k8s-app: filebeat
I was hoping that quick fix would do it but it looks like I'm gonna have to roll my sleeves up. I'll set up the same Filebeat importer on my system and get back to you in a bit. Either I will be able to get it working (as described above) or I will have to open a PR specifically for this use case. I will leave an update here either way 