If I want to scale my logstash (with PQ) horizontally in k8s, I would just increase the replica amount. Now that I have multiple nodes, I would like to understand how the PQ manages race conditions. I saw a .lock file but wasnt to sure if it answered all the use cases and additionally I was confused about why not use a volumeClaimTemplate (StatefulSet) for each replica/node therefore having no race condition.
I have one pipeline, but if I were to have more, there would be page.x and checkpoint files per pipeline or not, and if not so how do they manage race conditions?
I know this might be a nooby question, but I thank you regardless.
Is this the place then to raise a question about the bitnami helm chart not having pvc templates and sharing the main persistent volume between replicas/nodes?
Probably not, these helm charts are created and maintained by bitnami, so you need to ask on a bitnami forum or something like that.
I have no experience with k8s, but if by replica you mean another pod running logstash, then yes, it is a different logstash node/instance and the instances are independent from each other and need different persistent queues.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.