Multiple logstash concurrent data to the ES cluster, how to achieve data load balancing

Elastic Stack Logstash
1

Data is forwarded by logstash. How can I achieve load balancing to the ES cluster without causing data duplication?
Logstash can configure multiple output addresses, but how can we do data duplication? Is it using load balancing processing of ES clusters?

2

Set the hosts value .. it's a uri
It's all in the manual.
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-hosts

Value type is uri
Default value is [//127.0.0.1]
Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the hosts parameter. Remember the http protocol uses the http address (eg. 9200, not 9300). "127.0.0.1" ["127.0.0.1:9200","127.0.0.2:9200"] ["http://127.0.0.1"] ["https://127.0.0.1:9200"] ["https://127.0.0.1:9200/mypath"] (If using a proxy on a subpath) It is important to exclude dedicated master nodes from the hosts list to prevent LS from sending bulk requests to the master nodes. So this parameter should only reference either data or client nodes in Elasticsearch.

Any special characters present in the URLs here MUST be URL escaped! This means # should be put in as %23 for instance.

1 Like
3

thank you

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.