# K8s multiple replicas pq

**URL:** <https://discuss.elastic.co/t/k8s-multiple-replicas-pq/327887>\
**Category:** Logstash\
**Created:** [March 16, 2023, 9:42pm UTC](https://discuss.elastic.co/t/k8s-multiple-replicas-pq/327887 "2023-03-16T21:42:28Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![liel\_bondy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liel_bondy/32/118544_2.png) [@liel\_bondy](https://discuss.elastic.co/u/liel_bondy)\
**Post date:** [March 16, 2023, 9:42pm UTC](https://discuss.elastic.co/t/k8s-multiple-replicas-pq/327887/1 "2023-03-16T21:42:28Z")

</div>

Hey, quick question.

If I want to scale my logstash (with PQ) horizontally in k8s, I would just increase the replica amount. Now that I have multiple nodes, I would like to understand how the PQ manages race conditions. I saw a .lock file but wasnt to sure if it answered all the use cases and additionally I was confused about why not use a volumeClaimTemplate (StatefulSet) for each replica/node therefore having no race condition.

I have one pipeline, but if I were to have more, there would be page.x and checkpoint files per pipeline or not, and if not so how do they manage race conditions?

I know this might be a nooby question, but I thank you regardless.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 17, 2023, 1:08am UTC](https://discuss.elastic.co/t/k8s-multiple-replicas-pq/327887/2 "2023-03-17T01:08:06Z")

</div>

> [@liel\_bondy](#):
>
> If I want to scale my logstash (with PQ) horizontally in k8s, I would just increase the replica amount. Now that I have multiple nodes, I would like to understand how the PQ manages race conditions.

Logstash does not scale this way, persistent queues cannot be shared, each logstash needs its own persistent queue.

I do not use k8s, but each logstash node is independent from each other and you will need a different volume for each logstash node.

---

<div class="post-metadata">

**Author:** ![liel\_bondy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liel_bondy/32/118544_2.png) [@liel\_bondy](https://discuss.elastic.co/u/liel_bondy)\
**Post date:** [March 19, 2023, 9:36am UTC](https://discuss.elastic.co/t/k8s-multiple-replicas-pq/327887/3 "2023-03-19T09:36:01Z")

</div>

Is this the place then to raise a question about the bitnami helm chart not having pvc templates and sharing the main persistent volume between replicas/nodes?

Is it correct to regard to a replica as a node?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 19, 2023, 1:10pm UTC](https://discuss.elastic.co/t/k8s-multiple-replicas-pq/327887/4 "2023-03-19T13:10:15Z")

</div>

> [@liel\_bondy](#):
>
> Is this the place then to raise a question about the bitnami helm chart not having pvc templates and sharing the main persistent volume between replicas/nodes?

Probably not, these helm charts are created and maintained by bitnami, so you need to ask on a bitnami forum or something like that.

> [@liel\_bondy](#):
>
> Is it correct to regard to a replica as a node?

I have no experience with k8s, but if by replica you mean another pod running logstash, then yes, it is a different logstash node/instance and the instances are independent from each other and need different persistent queues.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 16, 2023, 1:11pm UTC](https://discuss.elastic.co/t/k8s-multiple-replicas-pq/327887/5 "2023-04-16T13:11:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
