Run multiple logstash replicas in Docker

[chmodrs (Mauricio Rocha)]

Hi,

I have an ELK cluster running in Docker Swarm mode. The cluster meed my needs because i have two replicas of elastic, but i have a problem with logstash running multiple instances.

When i send amount of logs to logstash, only one replica ingest data in elastic like bellow:

instance1 - 2 events ingested
instance2 - 100k events ingested

I allready try to use in memory queue and persistent queue, but get same problem.

Anyone have a idea for this problem? I'll need to use a kafka/redis in front of logstash?

Thanks!!

my logstash service in compose:

logstash:
command: logstash -f /usr/share/logstash/pipeline/logstash.conf
image: docker.elastic.co/logstash/logstash:6.6.0
volumes:
- "/opt/elk-swarm-cluster/logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml"
- "/opt/elk-swarm-cluster/logstash/pipeline/logstash.conf:/usr/share/logstash/pipeline/logstash.conf"
- "/opt/elk-swarm-cluster/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml"
ports:
- "5000:5000"
networks:
- elk
deploy:
mode: global
placement:
constraints: [node.role == worker]

my logstash.yml

http.host: "0.0.0.0"

xpack.monitoring.elasticsearch.url: http://elasticsearch

queue.type: persisted

path.queue: "/usr/share/logstash/data/queue"

queue.max_bytes: 3gb

[Badger]

What input do you have configured in logstash and what is sending data to it?

[chmodrs (Mauricio Rocha)]

Hi Bagder,

My logstash input

input {

 beats {

port => 5000

 }
}

logstash output

output {


 elasticsearch {

hosts => ["elasticsearch:9200","elasticsearch2:9200"]

action => "index"

template_name => "logs_company"

index => "%{nm_indice}_%{+YYYY_MM}"

 }

 stdout { }

}

I have a filebeat configured that sends nginx logs and my delphi application logs.

filebeat output

output.logstash:
  hosts: ["elb_loadbalancer_dns:5000"]

in front of my logstash i have an ELB configured with swarm master instances. All communication works, just logstash replicas don't work simultaneously.

Thanks!

[Badger]

The connection from filebeat to logstash is tcp. The load-balancer will establish a connection to one of the two instances and 100% of the traffic will go to that instance.

Using kafka would allow you to get traffic to go to both.

[chmodrs (Mauricio Rocha)]

The load balancer establish a connection with a docker logstash service (it contains two replicas), and the service should to distribute the traffic with two services. (theoretically)

Can be a problem with docker swarm routing or the solution is only use kafka?

I want to use kafka in future, the problem for use now is change the clients URL connection to my elb to kafka endpoint and i don't have knowledge in kafka

thanks for your help.

[Badger]

@chmodrs

The load balancer establish a connection with a docker logstash service (it contains two replicas), and the service should to distribute the traffic with two services. (theoretically)

Interesting. I have never seen a load balancer solution that works that way.

[Christian_Dahlqvist]

Beats use long-living connections, so once the connection has been established (especially as the load balancer makes it look like a single instance) all data will go through one of the instances. If you have lots of beats it may however even out over time as long as the loadbalancer do not prefer one instance over the other.

[chmodrs (Mauricio Rocha)]

Thanks for your help guys, i'll try to use multiple beats destination to resolve this issue, if not i'll use kafka in future

thanks