# Deploy a multi-replica Filebeat Deployment using PersistentVolumeClaims

**URL:** https://discuss.elastic.co/t/deploy-a-multi-replica-filebeat-deployment-using-persistentvolumeclaims/343610
**Category:** Elastic Cloud on Kubernetes (ECK)
**Created:** [September 22, 2023, 10:23am UTC](https://discuss.elastic.co/t/deploy-a-multi-replica-filebeat-deployment-using-persistentvolumeclaims/343610 "2023-09-22T10:23:14Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![niaomingjian](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niaomingjian/32/14844_2.png) [@niaomingjian](https://discuss.elastic.co/u/niaomingjian)
#### Post date: [September 22, 2023, 10:23am UTC](https://discuss.elastic.co/t/deploy-a-multi-replica-filebeat-deployment-using-persistentvolumeclaims/343610/1 "2023-09-22T10:23:14Z")

</div>

I want to use Filebeat to import data from a Kafka topic into Elasticsearch. For high reliability (so other pods can still work if one pod fails), I would like to deploy Filebeat as a multi-replica Deployment.

Deploy a multi-replica Filebeat Deployment using PersistentVolumeClaims

One of the pods fails to start.  
Log:

```auto
{"log.level":"error","@timestamp":"2023-09-22T10:20:18.868Z","log.origin":{"file.name":"instance/beat.go","file.line":1278},"message":"Exiting: /usr/share/filebeat/data/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)","service.name":"filebeat","ecs.version":"1.6.0"}

Exiting: /usr/share/filebeat/data/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)

```

Yaml:

```auto
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: quickstart
spec:
  type: filebeat
  version: 8.10.1
  elasticsearchRef:
    name: quickstart
  config:
    filebeat.inputs:
    - type: kafka
      hosts:
        - kafka730-cp-kafka-0.kafka730-cp-kafka-headless.one-namesapce:9092
        - kafka730-cp-kafka-1.kafka730-cp-kafka-headless.one-namesapce:9092
        - kafka730-cp-kafka-2.kafka730-cp-kafka-headless.one-namesapce:9092
      topics: ["my-topic"]
      group_id: "filebeat"
      initial_offsetedit: "newest"
      fields:
        runtime: "runtime-it"
      fields_under_root: true
      tags: ["it"]
    processors:
    - decode_json_fields:
        fields: ["message"]
        process_array: true
        max_depth: 1
        target: ""
        overwrite_keys: false
        add_error_key: true
  deployment:
    replicas: 2
    podTemplate:
      spec:
        volumes:
        - name: filebeat
          persistentVolumeClaim:
            claimName: filebeat
        securityContext:
          runAsUser: 0
        containers:
        - name: filebeat
          volumeMounts:
            - name: filebeat
              mountPath: /usr/share/filebeat/data

```

[Api Docs](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-api-beat-k8s-elastic-co-v1beta1.html#k8s-api-github-com-elastic-cloud-on-k8s-v2-pkg-apis-beat-v1beta1-deploymentspec) mentioned **`replicas`** field. So how can I use it?

---

<div class="post-metadata">

### Author: ![niaomingjian](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niaomingjian/32/14844_2.png) [@niaomingjian](https://discuss.elastic.co/u/niaomingjian)
#### Post date: [September 26, 2023, 1:24am UTC](https://discuss.elastic.co/t/deploy-a-multi-replica-filebeat-deployment-using-persistentvolumeclaims/343610/2 "2023-09-26T01:24:07Z")

</div>

In a [sample yaml](https://raw.githubusercontent.com/elastic/beats/8.10/deploy/kubernetes/filebeat-kubernetes.yaml), it says

> # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart

I'm using Kafka input, so I don't need a persistent volume to save read status for all files, and I can use `emptyDir` to overwrite it, right?

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 24, 2023, 1:24am UTC](https://discuss.elastic.co/t/deploy-a-multi-replica-filebeat-deployment-using-persistentvolumeclaims/343610/3 "2023-10-24T01:24:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
