# Run multiple logstash replicas in Docker **URL:** \ **Category:** Logstash\ **Created:** [February 15, 2019, 1:19pm UTC](https://discuss.elastic.co/t/run-multiple-logstash-replicas-in-docker/168595 "2019-02-15T13:19:32Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![chmodrs](https://avatars.discourse-cdn.com/v4/letter/c/a587f6/32.png) [@chmodrs](https://discuss.elastic.co/u/chmodrs)\ **Post date:** [February 15, 2019, 1:19pm UTC](https://discuss.elastic.co/t/run-multiple-logstash-replicas-in-docker/168595/1 "2019-02-15T13:19:32Z")
Hi, I have an ELK cluster running in Docker Swarm mode. The cluster meed my needs because i have two replicas of elastic, but i have a problem with logstash running multiple instances. When i send amount of logs to logstash, only one replica ingest data in elastic like bellow: instance1 - 2 events ingested instance2 - 100k events ingested I allready try to use in memory queue and persistent queue, but get same problem. Anyone have a idea for this problem? I'll need to use a kafka/redis in front of logstash? Thanks!! **my logstash service in compose:** logstash: command: logstash -f /usr/share/logstash/pipeline/logstash.conf image: [docker.elastic.co/logstash/logstash:6.6.0](http://docker.elastic.co/logstash/logstash:6.6.0) volumes: - "/opt/elk-swarm-cluster/logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml" - "/opt/elk-swarm-cluster/logstash/pipeline/logstash.conf:/usr/share/logstash/pipeline/logstash.conf" - "/opt/elk-swarm-cluster/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml" ports: - "5000:5000" networks: - elk deploy: mode: global placement: constraints: [node.role == worker] **my logstash.yml** http.host: "0.0.0.0" xpack.monitoring.elasticsearch.url: [http://elasticsearch](http://elasticsearch) queue.type: persisted path.queue: "/usr/share/logstash/data/queue" queue.max\_bytes: 3gb ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 15, 2019, 1:38pm UTC](https://discuss.elastic.co/t/run-multiple-logstash-replicas-in-docker/168595/2 "2019-02-15T13:38:50Z")
What input do you have configured in logstash and what is sending data to it? ---
**Author:** ![chmodrs](https://avatars.discourse-cdn.com/v4/letter/c/a587f6/32.png) [@chmodrs](https://discuss.elastic.co/u/chmodrs)\ **Post date:** [February 15, 2019, 1:45pm UTC](https://discuss.elastic.co/t/run-multiple-logstash-replicas-in-docker/168595/3 "2019-02-15T13:45:47Z")
Hi Bagder, **My logstash input** ``` input { beats { port => 5000 } } ``` **logstash output** ``` output { elasticsearch { hosts => ["elasticsearch:9200","elasticsearch2:9200"] action => "index" template_name => "logs_company" index => "%{nm_indice}_%{+YYYY_MM}" } stdout { } } ``` I have a filebeat configured that sends nginx logs and my delphi application logs. **filebeat output** ``` output.logstash: hosts: ["elb_loadbalancer_dns:5000"] ``` in front of my logstash i have an ELB configured with swarm master instances. All communication works, just logstash replicas don't work simultaneously. Thanks! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 15, 2019, 2:13pm UTC](https://discuss.elastic.co/t/run-multiple-logstash-replicas-in-docker/168595/4 "2019-02-15T14:13:25Z")
The connection from filebeat to logstash is tcp. The load-balancer will establish a connection to one of the two instances and 100% of the traffic will go to that instance. Using kafka would allow you to get traffic to go to both. ---
**Author:** ![chmodrs](https://avatars.discourse-cdn.com/v4/letter/c/a587f6/32.png) [@chmodrs](https://discuss.elastic.co/u/chmodrs)\ **Post date:** [February 15, 2019, 2:26pm UTC](https://discuss.elastic.co/t/run-multiple-logstash-replicas-in-docker/168595/5 "2019-02-15T14:26:46Z")
The load balancer establish a connection with a docker logstash service (it contains two replicas), and the service should to distribute the traffic with two services. (theoretically) Can be a problem with docker swarm routing or the solution is only use kafka? I want to use kafka in future, the problem for use now is change the clients URL connection to my elb to kafka endpoint and i don't have knowledge in kafka 😕 thanks for your help. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 15, 2019, 2:39pm UTC](https://discuss.elastic.co/t/run-multiple-logstash-replicas-in-docker/168595/6 "2019-02-15T14:39:06Z")
> [@chmodrs](#): > > The load balancer establish a connection with a docker logstash service (it contains two replicas), and the service should to distribute the traffic with two services. (theoretically) Interesting. I have never seen a load balancer solution that works that way. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 15, 2019, 2:52pm UTC](https://discuss.elastic.co/t/run-multiple-logstash-replicas-in-docker/168595/7 "2019-02-15T14:52:36Z")
Beats use long-living connections, so once the connection has been established (especially as the load balancer makes it look like a single instance) all data will go through one of the instances. If you have lots of beats it may however even out over time as long as the loadbalancer do not prefer one instance over the other. ---
**Author:** ![chmodrs](https://avatars.discourse-cdn.com/v4/letter/c/a587f6/32.png) [@chmodrs](https://discuss.elastic.co/u/chmodrs)\ **Post date:** [February 15, 2019, 3:31pm UTC](https://discuss.elastic.co/t/run-multiple-logstash-replicas-in-docker/168595/8 "2019-02-15T15:31:05Z")
Thanks for your help guys, i'll try to use multiple beats destination to resolve this issue, if not i'll use kafka in future thanks 😉 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 15, 2019, 3:31pm UTC](https://discuss.elastic.co/t/run-multiple-logstash-replicas-in-docker/168595/9 "2019-03-15T15:31:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.