Can't filebeat use multiple input/output config files in one instance?

Elastic Stack Beats
1

Want to deploy filebeat with 3 log definations together. Send to different output targets.

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:7.10.0
          args: [
            "-c", "/etc/logs1.yml",
            "-c", "/etc/logs2.yml",
            "-c", "/etc/logs3.yml",
            "-e",
          ]
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config-logs1
              mountPath: /etc/logs1.yml
              subPath: filebeat-logs1.yml
              readOnly: true
            - name: config-logs2
              mountPath: /etc/logs2.yml
              subPath: logs2.yml
              readOnly: true
            - name: config-logs3
              mountPath: /etc/logs3.yml
              subPath: logs3.yml
              readOnly: true
            - name: data
              mountPath: /usr/share/filebeat/data
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: config-logs1
          configMap:
            defaultMode: 0600
            name: configmap-logs1
        - name: config-logs2
          configMap:
            defaultMode: 0600
            name: configmap-logs2
        - name: config-logs3
          configMap:
            defaultMode: 0600
            name: configmap-logs3
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlog
          hostPath:
            path: /var/log
        - name: data
          hostPath:
            path: /var/lib/filebeat-data
            type: DirectoryOrCreate

logs1's configmap

data:
  filebeat-logs1.yml: |-
    filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - /var/log/logs1.json

    output.logstash:
      hosts: ["logstash-logs1.default.svc.cluster.local:5044"]

logs2's configmap

data:
  filebeat-logs2.yml: |-
    filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - /var/log/logs2.json

    output.logstash:
      hosts: ["logstash-logs2.default.svc.cluster.local:5044"]

logs3's configmap

data:
  filebeat-logs3.yml: |-
    filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - /var/log/logs3.json

    output.logstash:
      hosts: ["logstash-logs3.default.svc.cluster.local:5044"]

When each log files changed, every time it only send to the third logs3's output logstash-logs3.default.svc.cluster.local:5044. But can get data all of three logs1.json/logs2.json/logs3.json files.

Can't filebeat use multiple output in this case on one machine?

2

Do any of filebeats report any error in logs? Did you try to use config prospectors? See: Multiple configuration file with enabled option

3

I didn't find errors. Only work for the third output. I can use prospectores. This method seems can work: How to tag log files in filebeat for logstash ingestion?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.