Haproxy Passthrough Endpoints for ECE

balogan · January 27, 2020, 4:14pm

I'm attempting to setup haproxy to service all endpoints within an ECE deployment. First step with getting wildcard DNS setup is done. The problem is getting the endpoints passed through to the proxy servers. After googling I came up with this for a config but it's not working with attempts to open Kibana, Elasticsearch, or ES logs links in ECE timeout.

frontend fe_kib_es
bind *:9243
#acl is_wild hdr_dom(host) -m end .vcp-ecelab.mon.vzwops.com
acl is_wild hdr_dom(host) -i .vcp-ecelab.mon.vzwops.com
#use_backend be_kib_es if is_wild
use_backend be_kib_es if { hdr_end(host) -i .vcp-ecelab.mon.vzwops.com }

backend be_kib_es
mode http
cookie WILD_HTTP insert
balance source
option forwardfor header X-Forwarded-For
http-request set-header X-Real-IP %[src]
#http-request add-header X-Found-Cluster

server ecelab-kib-es-1 eceproxylab-1-southlake.mon.vzwops.com:9200 check verify none
server ecelab-kib-es-2 eceproxylab-2-southlake.mon.vzwops.com:9200 check verify none
server ecelab-kib-es-3 eceproxylab-3-southlake.mon.vzwops.com:9200 check verify none

Alex_Piggott · January 27, 2020, 5:05pm

When you say "ECE timeout", what does that mean exactly? What is the error code and what is the error string being returned?

Am I right that:

you are making a call to to $CLUSTER_ID.vcp-ecelab.mon.vzwops.com:9243 over https
haproxy is preserving the Host header (otherwise you'd need to set X-Found-Cluster) .. (if this was wrong you'd get a different error though I think)
eceproxylab-*-southlake.mon.vzwops.com:9200 is accessible from the haproxy box?
You are sure that use_backend be_kib_es if { hdr_end(host) -i .vcp-ecelab.mon.vzwops.com } is doing the right thing?

balogan · February 4, 2020, 4:49pm

What's timing out is my browser searching for endpoint.vcp-ecelab.mon.vzwops.com:9243

you are making a call to to $CLUSTER_ID.vcp-ecelab.mon.vzwops.com:9243 over https
Correct
haproxy is preserving the Host header (otherwise you'd need to set X-Found-Cluster ) .. (if this was wrong you'd get a different error though I think)
I believe this is correct
eceproxylab-*-southlake.mon.vzwops.com:9200 is accessible from the haproxy box?
Correct
You are sure that use_backend be_kib_es if { hdr_end(host) -i .vcp-ecelab.mon.vzwops.com } is doing the right thing?
Not sure

Today I made some changes after reading the haproxy doesn't strip headers and haproxy allowed passing of the endpoints, but it broke the rule for the platform UI because ES, Kibana, and platform UI use the same domain name and forward to different locations. Here's what I did.

use_backend be_ecelab if { hdr_end(host) -i vcp-ecelab.mon.vzwops.com }

backend be_ecelab
  mode http
  balance source
  option forwardfor header X-Forwarded-For
  http-request set-header X-Real-IP %[src]

  server ecelab-kib-1 eceproxylab-1-southlake.mon.vzwops.com:9200 check verify none
  server ecelab-kib-2 eceproxylab-2-southlake.mon.vzwops.com:9200 check verify none
  server ecelab-kib-3 eceproxylab-3-southlake.mon.vzwops.com:9200 check verify none

  #server ecelab-1 ecedirectorlab-1-southlake.mon.vzwops.com:12400 check verify none
  #server ecelab-2 ecedirectorlab-2-southlake.mon.vzwops.com:12400 check verify none
  #server ecelab-3 ecedirectorlab-3-southlake.mon.vzwops.com:12400 check verify none

So my question is how to do I get both the platform UI and other endpoints that forward to different locations (directors vs. ECE proxies) to coexist?

Alex_Piggott · February 4, 2020, 5:43pm

To handle UI/API differently from proxies, can you have another rule, which matches on the host being exactly vcp-ecelab.mon.vzwops.com ... you could also reason on port if you're not mapping everything to 443 (eg 12443 vs 9243)

I had a hunt around and found this sample haproxy LB config, in case it helps:

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private
	ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
	ssl-default-bind-options no-sslv3

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
        timeout connect 5000
        timeout client 310000ms
        timeout server 310000ms
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http   

# Admin console LB
frontend haproxy-adminconsole
        bind <HAPROXYHOST>:12400
        default_backend adminconsole

backend adminconsole
        balance roundrobin
        server adminconsole-1 <HOST_ECE_1>:12400 check
        server adminconsole-2 <HOST_ECE_2>:12400 check
        server adminconsole-3 <HOST_ECE_3>:12400 check

frontend haproxy-adminconsole-ssl
        bind <HAPROXYHOST>:12443
        mode tcp
        option tcplog
        option ssl-hello-chk        
        default_backend haproxy-adminconsole-ssl

backend haproxy-adminconsole-ssl
        mode tcp
        option tcplog
        balance roundrobin        
        server haproxy-adminconsole-ssl-1 <HOST_ECE_1>:12443 check
        server haproxy-adminconsole-ssl-2 <HOST_ECE_2>:12443 check
        server haproxy-adminconsole-ssl-3 <HOST_ECE_3>:12443 check        


# Proxy LB

frontend haproxy-proxy
        option httpchk GET /__elb_health__   
        bind <HAPROXYHOST>:9200
        default_backend proxy

backend proxy
        balance roundrobin
        server proxy-1 <HOST_ECE_1>:9200 check port 9200
        server proxy-2 <HOST_ECE_2>:9200 check port 9200
        server proxy-3 <HOST_ECE_3>:9200 check port 9200


frontend haproxy-proxy-ssl
        mode tcp
        option tcplog
        option httpchk GET /__elb_health__
        option ssl-hello-chk
        bind <HAPROXYHOST>:9243
        default_backend proxy-ssl

backend proxy-ssl
        balance roundrobin
        mode tcp
        option tcplog        
        server proxy-ssl-1 <HOST_ECE_1>:9243 check port 9200
        server proxy-ssl-2 <HOST_ECE_2>:9243 check port 9200
        server proxy-ssl-3 <HOST_ECE_3>:9243 check port 9200 


listen stats 
        bind *:9000
        mode http
        balance
        stats enable
        stats uri /stats
        stats realm HAProxy\ Statistics
        stats auth admin:password-haproxy
        stats admin if TRUE

balogan · February 5, 2020, 2:59pm

I was able to get through haproxy with your example using port 9200 on the client side, thanks. Unfortunately port 9243 is blocked at our firewall, it takes an act of God to get a port opened, and as you know Kibana requires SSL for authentication. Is it possible to get Kibana to authenticate a client using port 9200?

Alex_Piggott · February 6, 2020, 9:18pm

What exactly do you want your port settings to look like? Is it

browser -> hostname:9243 -> haproxy -> firewall -> kibana:9200

?

And the problem is that firewall already lets through 9200 but not 9243?

Are you saying you are seeing problems with hitting kibana:9200 over HTTP? Because that should work fine, I think (you're actually hitting a proxy service that picks a Kibana instance and hits that over HTTPS)

balogan · February 7, 2020, 12:57pm

The problem is browser -> hostname:9243 -> haproxy that's where it's blocked.

And the problem is that firewall already lets through 9200 but not 9243?
Exactly

Yesterday I tried putting 'server.ssl.enabled: false' in kibana.yml via the UI and after a restart of the Kibana nodes they didn't come up healthy.

Alex_Piggott · February 7, 2020, 3:25pm

Yesterday I tried putting 'server.ssl.enabled: false' in kibana.yml via the UI

Don't do this ... the ECE-side architecture looks like:

from_client -> ece-proxy-service:9200 -> (adds internal cert) -> kibana:https

So regardless of whether you specify 9200 or 9443 in your client, the actual transaction between the ECE proxy and Kibana will always be over SSL

Have you definitely confirmed that with server.ssl.enabled left alone that using 9200 fails (and if so with what error?)

balogan · February 7, 2020, 5:33pm

Ok good to know. I'll take this to my manager and try to get 9243 punched in the FW.

Yeah when using port 9200 Kibana complains stating SSL is required to login. I don't remember the exact wording.

Thanks again for the help.

Alex_Piggott · February 7, 2020, 6:11pm

If you have a moment I'd love to see the 9200 request (+ what you believe the haproxy maps it to) and the response from Kibana ... everyone here is scratching their heads over how it could happen

balogan · February 7, 2020, 7:10pm

I just rebuilt the environment today and I need it up to show some things to my manager, so maybe Monday I can get around to reproducing the issue.

system · February 21, 2020, 7:10pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.