K8s multiple replicas pq

Elastic Stack Logstash
1

Hey, quick question.

If I want to scale my logstash (with PQ) horizontally in k8s, I would just increase the replica amount. Now that I have multiple nodes, I would like to understand how the PQ manages race conditions. I saw a .lock file but wasnt to sure if it answered all the use cases and additionally I was confused about why not use a volumeClaimTemplate (StatefulSet) for each replica/node therefore having no race condition.

I have one pipeline, but if I were to have more, there would be page.x and checkpoint files per pipeline or not, and if not so how do they manage race conditions?

I know this might be a nooby question, but I thank you regardless.

2

Logstash does not scale this way, persistent queues cannot be shared, each logstash needs its own persistent queue.

I do not use k8s, but each logstash node is independent from each other and you will need a different volume for each logstash node.

1 Like
3

Is this the place then to raise a question about the bitnami helm chart not having pvc templates and sharing the main persistent volume between replicas/nodes?

Is it correct to regard to a replica as a node?

4

Probably not, these helm charts are created and maintained by bitnami, so you need to ask on a bitnami forum or something like that.

I have no experience with k8s, but if by replica you mean another pod running logstash, then yes, it is a different logstash node/instance and the instances are independent from each other and need different persistent queues.