Elasticsearch output - load balancing selection

kobyb · September 29, 2016, 7:04pm

Say I have three logstash servers loadbalanced behind and F5. All three servers are identical, and contain an elasticsearch output plugin with a host array of something like: hosts => ["host1","host2",host3"].

How is the output load balanced? Round Robin, least used, etc? What I'm trying to figure out is, if the three logstash servers would get in a rhythm of rotating through the same hosts at the same time, thus swamping one, while the others go unused.

Thanks

magnusbaeck · October 5, 2016, 6:01am

The way I read the code (below) each Logstash instance will distribute the requests evenly between the hosts and for hosts with an equal number of posted requests it'll pick one of them at random. So I guess least used followed by round robin as a tie-breaker.

github.com

logstash-plugins/logstash-output-elasticsearch/blob/v5.1.2/lib/logstash/outputs/elasticsearch/http_client/pool.rb#L345-L371


def get_connection
  @state_mutex.synchronize do
    # The goal here is to pick a random connection from the least-in-use connections
    # We want some randomness so that we don't hit the same node over and over, but
    # we also want more 'fair' behavior in the event of high concurrency
    eligible_set = nil
    lowest_value_seen = nil
    @url_info.each do |url,meta|
      meta_in_use = meta[:in_use]
      next if meta[:dead]


      if lowest_value_seen.nil? || meta_in_use < lowest_value_seen
        lowest_value_seen = meta_in_use
        eligible_set = [[url, meta]]
      elsif lowest_value_seen == meta_in_use
        eligible_set << [url, meta]
      end
    end


    return nil if eligible_set.nil?

This file has been truncated. show original

system · July 6, 2017, 4:35am