I am using below structure of event processing pipeline in my project.
LSF-->logstash-shipper(Scaled)--->Redis--->Indexer(Scaled)--->NGINX--->Multiple nodes ES
Here, I want to make Shipper and indexer highly available.I was referring to below link https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html
I have doubt in diagram 7 on the page.
I am not clear how load balancing is achieved here? Its scaling what has shown in the diagram. What takes care of load balancing in this diagram? Like in my case nGinx's upstream declaration load balances ES cluster and also prevents duplicates.
Can anybody please elaborate diagram and answer my questions?
If LSF1 ships logs to shipper-instance-1 then how can I make this instance highly available?
in case of failure of instance I want to make available similar instance which takes logs from LSF1 but never creates duplicate messages in messageQ. How this can be achieved?