I want to add 2 logstash in my application for scalability.From availablity point... I have a question does 2 logstash works simultaneously or they works as a active-passive (one stop other takes) or they automatically manage the load and yes i have multiple pipelines id.
logstash does not support clustering. If you have two copies of logstash they will both process their inputs. For a tcp or http input you could put a load balancer in front of them and each request would get processed in one of the two. For a file input they would both read the file and all of the data would be duplicated.
I want to know that if i have some logstash pipeline files do i need to keep these logstash pipeline files in both the nodes..if i do this then i have duplicate data in elasticsearch and the reason i want to use 2 logstash is for high availability but it seems that it's not good idea to have 2 logstash
I have 5 logstash instances that pull from 3 Redis servers so it will never create duplicate documents. If you're concerned about the same file data going to each logstash then create a unique document ID. GL
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.