hi. I'm collecting my k8s containers all logs in /var/log/containers/*.log
and I need to parsing with modules(e.g. kafka, zookeeper, mongodb etc.)
but I cannot find reference.
I already know about way like 'how to set up filebeat in elastic cloud on kubernetes' and filebeat reference, filebeat module reference etc.
but cannot apply filebeat module on my k8s filebeat.
this is my filebeat configuration.
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: filebeat
spec:
type: filebeat
version: 8.2.0
elasticsearchRef:
name: elasticsearch
kibanaRef:
name: kibana
config:
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
default_matchers.enabled: false
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: /var/log/containers/
filebeat.modules:
- module: kafka
log:
enabled: true
filebeat.config.modules:
path: usr/share/filebeat/modules.d/*.yml
enabled: true
processors:
- fingerprint:
fields: ['@timestamp', 'log.file.path', 'log.offset']
target_field: '@metadata._id'
daemonSet:
podTemplate:
spec:
serviceAccountName: filebeat
automountServiceAccountToken: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
securityContext:
runAsUser: 0
containers:
- name: filebeat
volumeMounts:
- name: varlogcontainers
mountPath: /var/log/containers
- name: varlogpods
mountPath: /var/log/pods
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
- name: kafkamodule
mountPath: /usr/share/filebeat/modules.d
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumes:
- name: varlogcontainers
hostPath:
path: /var/log/containers
- name: varlogpods
hostPath:
path: /var/log/pods
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: kafkamodule
configMap:
name: filebeat-kafka
items:
- key: kafkamodule
path: kafka.yml
- key: zookeepermodule
path: zookeeper.yml
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-kafka
data:
kafkamodule: |-
- module: kafka
log:
enabled: true
var.paths:
- '/var/log/containers/kafka-*.log'
zookeepermodule: |-
- module: zookeeper
log:
enabled: true
var.paths:
- '/var/log/containers/*zookeeper*.log'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: filebeat
rules:
- apiGroups: [''] # "" indicates the core API group
resources:
- namespaces
- pods
- nodes
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: default
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
It always return error "Exiting: module kafka is configured but has no enabled filesets"
-> I fix it as filebeat.modules setting log.enabled=true, but it doesn't work.
return "message":"Enabled modules/filesets: kafka (log)"
exclude filebeat.modules, active normally, but module not applied.
(cannot find in kibana discover filebeat field)
how to solve it?