# Filebeat modules with filebeat input on ECK

**URL:** <https://discuss.elastic.co/t/filebeat-modules-with-filebeat-input-on-eck/305034>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Tags:** docker\
**Created:** [May 18, 2022, 8:24am UTC](https://discuss.elastic.co/t/filebeat-modules-with-filebeat-input-on-eck/305034 "2022-05-18T08:24:41Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ryuseongryong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryuseongryong/32/98889_2.png) [@ryuseongryong](https://discuss.elastic.co/u/ryuseongryong)\
**Post date:** [May 18, 2022, 8:24am UTC](https://discuss.elastic.co/t/filebeat-modules-with-filebeat-input-on-eck/305034/1 "2022-05-18T08:24:41Z")

</div>

hi. I'm collecting my k8s containers all logs in `/var/log/containers/*.log`  
and I need to parsing with modules(e.g. kafka, zookeeper, mongodb etc.)

but I cannot find reference.  
I already know about way like 'how to set up filebeat in elastic cloud on kubernetes' and filebeat reference, filebeat module reference etc.  
but cannot apply filebeat module on my k8s filebeat.

this is my filebeat configuration.

```auto
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
spec:
  type: filebeat
  version: 8.2.0
  elasticsearchRef:
    name: elasticsearch
  kibanaRef:
    name: kibana
  config:
    filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
          - add_kubernetes_metadata:
              default_matchers.enabled: false
              host: ${NODE_NAME}
              matchers:
                - logs_path:
                    logs_path: /var/log/containers/
        
    filebeat.modules:
      - module: kafka
        log:
          enabled: true
    filebeat.config.modules:
      path: usr/share/filebeat/modules.d/*.yml
      enabled: true

    processors:
      - fingerprint:
          fields: ['@timestamp', 'log.file.path', 'log.offset']
          target_field: '@metadata._id'
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        securityContext:
          runAsUser: 0
        containers:
          - name: filebeat
            volumeMounts:
              - name: varlogcontainers
                mountPath: /var/log/containers
              - name: varlogpods
                mountPath: /var/log/pods
              - name: varlibdockercontainers
                mountPath: /var/lib/docker/containers
              - name: kafkamodule
                mountPath: /usr/share/filebeat/modules.d

            env:
              - name: NODE_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: spec.nodeName
        volumes:
          - name: varlogcontainers
            hostPath:
              path: /var/log/containers
          - name: varlogpods
            hostPath:
              path: /var/log/pods
          - name: varlibdockercontainers
            hostPath:
              path: /var/lib/docker/containers
          - name: kafkamodule
            configMap:
              name: filebeat-kafka
              items:
                - key: kafkamodule
                  path: kafka.yml
                - key: zookeepermodule
                  path: zookeeper.yml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-kafka
data:
  kafkamodule: |-
    - module: kafka
      log:
        enabled: true
        var.paths:
          - '/var/log/containers/kafka-*.log'
  zookeepermodule: |-
    - module: zookeeper
      log:
        enabled: true
        var.paths:
          - '/var/log/containers/*zookeeper*.log'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
rules:
  - apiGroups: [''] # "" indicates the core API group
    resources:
      - namespaces
      - pods
      - nodes
    verbs:
      - get
      - watch
      - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: default
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

```

It always return error `"Exiting: module kafka is configured but has no enabled filesets"`  
-\> I fix it as filebeat.modules setting log.enabled=true, but it doesn't work.  
return `"message":"Enabled modules/filesets: kafka (log)"`

exclude filebeat.modules, active normally, but module not applied.  
(cannot find in kibana discover filebeat field)

how to solve it?

---

<div class="post-metadata">

**Author:** ![ryuseongryong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryuseongryong/32/98889_2.png) [@ryuseongryong](https://discuss.elastic.co/u/ryuseongryong)\
**Post date:** [May 18, 2022, 8:29am UTC](https://discuss.elastic.co/t/filebeat-modules-with-filebeat-input-on-eck/305034/2 "2022-05-18T08:29:09Z")

</div>

I've also tried using the command line and it didn't work.

```auto
command: ["bash", "-c"]
args: ["./filebeat modules enable kafka; ./filebeat setup -e; ./filebeat -e]

```

---

<div class="post-metadata">

**Author:** ![ryuseongryong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryuseongryong/32/98889_2.png) [@ryuseongryong](https://discuss.elastic.co/u/ryuseongryong)\
**Post date:** [May 19, 2022, 3:19am UTC](https://discuss.elastic.co/t/filebeat-modules-with-filebeat-input-on-eck/305034/3 "2022-05-19T03:19:00Z")

</div>

maybe use as autodiscover

---

<div class="post-metadata">

**Author:** ![ryuseongryong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryuseongryong/32/98889_2.png) [@ryuseongryong](https://discuss.elastic.co/u/ryuseongryong)\
**Post date:** [May 19, 2022, 6:58am UTC](https://discuss.elastic.co/t/filebeat-modules-with-filebeat-input-on-eck/305034/4 "2022-05-19T06:58:24Z")

</div>

kafka version mismatch.... maybe just use dissect message

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 16, 2022, 6:58am UTC](https://discuss.elastic.co/t/filebeat-modules-with-filebeat-input-on-eck/305034/5 "2022-06-16T06:58:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.