I want to use Filebeat to import data from a Kafka topic into Elasticsearch. For high reliability (so other pods can still work if one pod fails), I would like to deploy Filebeat as a multi-replica Deployment.
Deploy a multi-replica Filebeat Deployment using PersistentVolumeClaims
One of the pods fails to start.
Log:
{"log.level":"error","@timestamp":"2023-09-22T10:20:18.868Z","log.origin":{"file.name":"instance/beat.go","file.line":1278},"message":"Exiting: /usr/share/filebeat/data/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: /usr/share/filebeat/data/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)
Yaml:
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: quickstart
spec:
type: filebeat
version: 8.10.1
elasticsearchRef:
name: quickstart
config:
filebeat.inputs:
- type: kafka
hosts:
- kafka730-cp-kafka-0.kafka730-cp-kafka-headless.one-namesapce:9092
- kafka730-cp-kafka-1.kafka730-cp-kafka-headless.one-namesapce:9092
- kafka730-cp-kafka-2.kafka730-cp-kafka-headless.one-namesapce:9092
topics: ["my-topic"]
group_id: "filebeat"
initial_offsetedit: "newest"
fields:
runtime: "runtime-it"
fields_under_root: true
tags: ["it"]
processors:
- decode_json_fields:
fields: ["message"]
process_array: true
max_depth: 1
target: ""
overwrite_keys: false
add_error_key: true
deployment:
replicas: 2
podTemplate:
spec:
volumes:
- name: filebeat
persistentVolumeClaim:
claimName: filebeat
securityContext:
runAsUser: 0
containers:
- name: filebeat
volumeMounts:
- name: filebeat
mountPath: /usr/share/filebeat/data
Api Docs mentioned replicas field. So how can I use it?