Filebeat modules with filebeat input on ECK

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

hi. I'm collecting my k8s containers all logs in /var/log/containers/*.log
and I need to parsing with modules(e.g. kafka, zookeeper, mongodb etc.)

but I cannot find reference.
I already know about way like 'how to set up filebeat in elastic cloud on kubernetes' and filebeat reference, filebeat module reference etc.
but cannot apply filebeat module on my k8s filebeat.

this is my filebeat configuration.

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
spec:
  type: filebeat
  version: 8.2.0
  elasticsearchRef:
    name: elasticsearch
  kibanaRef:
    name: kibana
  config:
    filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
          - add_kubernetes_metadata:
              default_matchers.enabled: false
              host: ${NODE_NAME}
              matchers:
                - logs_path:
                    logs_path: /var/log/containers/
        
    filebeat.modules:
      - module: kafka
        log:
          enabled: true
    filebeat.config.modules:
      path: usr/share/filebeat/modules.d/*.yml
      enabled: true

    processors:
      - fingerprint:
          fields: ['@timestamp', 'log.file.path', 'log.offset']
          target_field: '@metadata._id'
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        securityContext:
          runAsUser: 0
        containers:
          - name: filebeat
            volumeMounts:
              - name: varlogcontainers
                mountPath: /var/log/containers
              - name: varlogpods
                mountPath: /var/log/pods
              - name: varlibdockercontainers
                mountPath: /var/lib/docker/containers
              - name: kafkamodule
                mountPath: /usr/share/filebeat/modules.d

            env:
              - name: NODE_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: spec.nodeName
        volumes:
          - name: varlogcontainers
            hostPath:
              path: /var/log/containers
          - name: varlogpods
            hostPath:
              path: /var/log/pods
          - name: varlibdockercontainers
            hostPath:
              path: /var/lib/docker/containers
          - name: kafkamodule
            configMap:
              name: filebeat-kafka
              items:
                - key: kafkamodule
                  path: kafka.yml
                - key: zookeepermodule
                  path: zookeeper.yml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-kafka
data:
  kafkamodule: |-
    - module: kafka
      log:
        enabled: true
        var.paths:
          - '/var/log/containers/kafka-*.log'
  zookeepermodule: |-
    - module: zookeeper
      log:
        enabled: true
        var.paths:
          - '/var/log/containers/*zookeeper*.log'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
rules:
  - apiGroups: [''] # "" indicates the core API group
    resources:
      - namespaces
      - pods
      - nodes
    verbs:
      - get
      - watch
      - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: default
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

It always return error "Exiting: module kafka is configured but has no enabled filesets"
-> I fix it as filebeat.modules setting log.enabled=true, but it doesn't work.
return "message":"Enabled modules/filesets: kafka (log)"

exclude filebeat.modules, active normally, but module not applied.
(cannot find in kibana discover filebeat field)

how to solve it?

2

I've also tried using the command line and it didn't work.

command: ["bash", "-c"]
args: ["./filebeat modules enable kafka; ./filebeat setup -e; ./filebeat -e]
3

maybe use as autodiscover

4

kafka version mismatch.... maybe just use dissect message

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.