K8s multiple replicas pq

Hey, quick question.

If I want to scale my logstash (with PQ) horizontally in k8s, I would just increase the replica amount. Now that I have multiple nodes, I would like to understand how the PQ manages race conditions. I saw a .lock file but wasnt to sure if it answered all the use cases and additionally I was confused about why not use a volumeClaimTemplate (StatefulSet) for each replica/node therefore having no race condition.

I have one pipeline, but if I were to have more, there would be page.x and checkpoint files per pipeline or not, and if not so how do they manage race conditions?

I know this might be a nooby question, but I thank you regardless.

Logstash does not scale this way, persistent queues cannot be shared, each logstash needs its own persistent queue.

I do not use k8s, but each logstash node is independent from each other and you will need a different volume for each logstash node.

1 Like

Is this the place then to raise a question about the bitnami helm chart not having pvc templates and sharing the main persistent volume between replicas/nodes?

Is it correct to regard to a replica as a node?

Probably not, these helm charts are created and maintained by bitnami, so you need to ask on a bitnami forum or something like that.

I have no experience with k8s, but if by replica you mean another pod running logstash, then yes, it is a different logstash node/instance and the instances are independent from each other and need different persistent queues.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.