Logstash and elasticsearch behind HA-proxy

[pdesoyres (Pierre De Soyres)]

Hello,

I am trying to output events to elasticsearch :

input {
    heartbeat {
        interval => 5
            message  => 'Hello from Logstash'
    }
}
output {elasticsearch {hosts => [ 'http://registry.marathon.mesos:10004' ]}  }

elasticsearch is behind HA-proxy. (If I don't put elasticsearch behind HA-proxy, everything works fine)

I'm getting the following errors in logstash logs :

Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties.
[2016-11-21T14:50:07,254][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>["http://registry.marathon.mesos:10004"]}}
[2016-11-21T14:50:07,552][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://registry.marathon.mesos:10004"]}
[2016-11-21T14:50:07,565][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000}
[2016-11-21T14:50:07,575][INFO ][logstash.pipeline        ] Pipeline main started
[2016-11-21T14:50:07,666][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2016-11-21T14:50:24,191][WARN ][logstash.outputs.elasticsearch] Marking url as dead. {:reason=>"Elasticsearch Unreachable: [http://registry.marathon.mesos:10004][Manticore::ClientProtocolException] registry.marathon.mesos:10004 failed to respond", :url=>#<URI::HTTP:0x365002a7 URL:http://registry.marathon.mesos:10004>, :error_message=>"Elasticsearch Unreachable: [http://registry.marathon.mesos:10004][Manticore::ClientProtocolException] registry.marathon.mesos:10004 failed to respond", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2016-11-21T14:50:24,193][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://registry.marathon.mesos:10004][Manticore::ClientProtocolException] registry.marathon.mesos:10004 failed to respond", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>4}
[2016-11-21T14:50:27,135][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
[2016-11-21T14:50:27,136][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>2}
[2016-11-21T14:50:27,500][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:url=>#<URI::HTTP:0xe3cb2c4 URL:http://registry.marathon.mesos:10004>, :healthcheck_path=>"/"}
[2016-11-21T14:50:27,508][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0xe3cb2c4 URL:http://registry.marathon.mesos:10004>}

HA-proxy configuration is :

frontend elasticsearch_10004
  bind *:10004
  mode http
  use_backend elasticsearch_10004
frontend elasticsearch_10005
  bind *:10005
  mode tcp
  use_backend elasticsearch_10005
backend elasticsearch_10004
  balance roundrobin
  mode http
  option forwardfor
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request add-header X-Forwarded-Proto https if { ssl_fc }
  option  httpchk GET /_cat/health
  timeout check 5s
  server 192_168_0_30_29130 192.168.0.30:29130 check inter 10s fall 4
backend elasticsearch_10005
  balance roundrobin
  mode tcp
  server 192_168_0_30_29131 192.168.0.30:29131

I can see the bulk requests going through HA-proxy :

haproxy[13751]: 172.17.0.7:57470 [21/Nov/2016:13:41:33.268] elasticsearch_10004 elasticsearch_10004/192_168_0_30_29130 408/0/0/5/413 200 332 - - ---- 2/2/0/1/0 0/0 "POST /_bulk HTTP/1.1"

I can see bulk request handled by elasticsearch (I set the elasticsearch log level to TRACE to see them in log file)
If I manually bulk (using curl as describe in official documentation), It works.

I'm using version 5.0.0

Could anyone figure out what is going on ?

[pdesoyres (Pierre De Soyres)]

I forgot to say that the events are fully indexed in elasticsearch. I just want to know why a get those errors in logstash when going through HA-proxy.

[Andrew_Cholakian1 (Andrew Cholakian)]

@pdesoyres two thoughts:

1. Why are you using HA Proxy? Why not just use the ES output with sniffing enabled? Logstash load balances across auto-scaling ES clusters just fine.
2. To debug this I'd recommend hauling out tcpdump or wireshark to do packet captures to figure out what's going on.

Looking at the logs, it looks like the ES output healthcheck to / is succeeding, but the actual bulk request is failing. I'm not sure what's happening at HAProxy, but would love to find out.