Load balance output between multiple hosts

Elastic Stack Logstash
1

Hi,
I have a logstash instance with multiple hosts output. According to the documentation, logstash should load balance across the hosts specified. However, I am receiving the logs on both of the elasticsearch instance.

Am I doing something wrong and how should I be configuring it? Below is a sample of my configuration. Thanks in advance.

output {
elasticsearch {
hosts => ["x.x.x.x:9200", "y.y.y.y:9200"]
}
}

2

How are you determining this? Are you seeing duplicates?

3

Hi,

I'm not sure if this is the correct approach but I configured it to be two single-node cluster so as to see which elasticsearch node will receive the logs.

4

And what are you seeing? Do all logs get indexed into both instances?

5

Yup, the logs appears in both of the instances.

6

Can you give the output of the cluster stats API from both nodes?

7

Hi there,

Sorry for the late reply. I have changed my configuration file and have noticed that the output now only goes to one instance at a time. I was just wondering if this is just a coincidence or does the "start_position" in the input plugin affects this?
i.e. I changed start_position from "beginning" to "end".

Also, may I ask how the load balancing is done? E.g. is it round robin or least used?

As for the cluster stats API, here are the links to the pastebin:
Node1: https://pastebin.com/1TQu4Q0k
Node2: https://pastebin.com/MjSwGJwh

Once again, sorry for the late reply and thank you in advance!

Edit: Replaced pastebin url as it expired.

8

bump

9

This affects how Logstash reads files, not where the data is sent.

Am not sure.

10

That is my thought too. However, that was the only part that I changed during my testing, so I'm still not really sure how the load balancing is done.

Okay, thank you so much for the help though! :slight_smile:

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.