Deploy a multi-replica Filebeat Deployment using PersistentVolumeClaims

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I want to use Filebeat to import data from a Kafka topic into Elasticsearch. For high reliability (so other pods can still work if one pod fails), I would like to deploy Filebeat as a multi-replica Deployment.

Deploy a multi-replica Filebeat Deployment using PersistentVolumeClaims

One of the pods fails to start.
Log:

{"log.level":"error","@timestamp":"2023-09-22T10:20:18.868Z","log.origin":{"file.name":"instance/beat.go","file.line":1278},"message":"Exiting: /usr/share/filebeat/data/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)","service.name":"filebeat","ecs.version":"1.6.0"}

Exiting: /usr/share/filebeat/data/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)

Yaml:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: quickstart
spec:
  type: filebeat
  version: 8.10.1
  elasticsearchRef:
    name: quickstart
  config:
    filebeat.inputs:
    - type: kafka
      hosts:
        - kafka730-cp-kafka-0.kafka730-cp-kafka-headless.one-namesapce:9092
        - kafka730-cp-kafka-1.kafka730-cp-kafka-headless.one-namesapce:9092
        - kafka730-cp-kafka-2.kafka730-cp-kafka-headless.one-namesapce:9092
      topics: ["my-topic"]
      group_id: "filebeat"
      initial_offsetedit: "newest"
      fields:
        runtime: "runtime-it"
      fields_under_root: true
      tags: ["it"]
    processors:
    - decode_json_fields:
        fields: ["message"]
        process_array: true
        max_depth: 1
        target: ""
        overwrite_keys: false
        add_error_key: true
  deployment:
    replicas: 2
    podTemplate:
      spec:
        volumes:
        - name: filebeat
          persistentVolumeClaim:
            claimName: filebeat
        securityContext:
          runAsUser: 0
        containers:
        - name: filebeat
          volumeMounts:
            - name: filebeat
              mountPath: /usr/share/filebeat/data

Api Docs mentioned replicas field. So how can I use it?

2

In a sample yaml, it says

data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart

I'm using Kafka input, so I don't need a persistent volume to save read status for all files, and I can use emptyDir to overwrite it, right?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.