Kibana Endpoint not Passing through Haproxy Using Shortened URL

[balogan (Brett Logan)]

ECE 2.4.3
Haproxy 1.8
Kibana/Elasticsearch 7.5.4

I can get to Kibana using the endpoint prepended to the domain name like this:
https://708....vcp-ecelab-log.mon.vzwops.com

But we want a more user friendly url like this:
https://vcp-ecelab-log.mon.vzwops.com/kibana

Here's the result:
{"ok":false,"message":"Unknown deployment."}

Haproxy setup:
Frontend:
use_backend be_ecelab_log if { hdr(host) -i vcp-ecelab-log.mon.vzwops.com } { path_beg -i /kibana } or { path_beg -i /kibana/ }

Backend:
backend be_ecelab_log
mode http
balance source

server ecelab-log-1 708.....eceproxylab-1-southlake.mon.vzwops.com:9200 check verify none
server ecelab-log-2 708.....eceproxylab-2-southlake.mon.vzwops.com:9200 check verify none
server ecelab-log-3 708.....eceproxylab-3-southlake.mon.vzwops.com:9200 check verify none

I've tried different combinations of the following on the frontend with no luck:
#acl ece_kibana path_beg -i /kibana
#redirect location 708.....eceproxylab-1-southlake.mon.vzwops.com append-slash code 301 if ece_kibana
#acl ece_kibana { hdr(host) -i vcp-ecelab-log.mon.vzwops.com } { path_beg -i /kibana } or { path_beg -i /kibana/ }
#http-request set-var(req.kibana_endpoint) req.hdr(host),lower,regsub(.vcp-ecelab-log.mon.vzwops.com$,) if { hdr(host) -i vcp-ecelab-log.mon.vzwops.com } { path_beg -i /kibana }
#http-request set-path /%[var(req.rewrite_kibendpoint)]%[path] if { var(req.rewrite_kibendpoint) -m found }
#http-request set-header Host vcp-ecelab-log.mon.vzwops.com if { var(req.rewrite_kibendpoint) -m found }
#http-request redirect location 708.....eceproxylab-1-southlake.mon.vzwops.com if ece_kibana
#http-request redirect code 301 location http://%[url,regsub(^/,/708....,)].%[hdr(host)] if ece_kibana
#use_backend be_ecelab_log ece_kibana
#redirect prefix 708.....eceproxylab-1-southlake.mon.vzwops.com code 301 if { hdr(host) -i vcp-ecelab-log.mon.vzwops.com }

[Alex_Piggott]

Hi @balogan

We're working on built-in support for "vanity URLs" aka "cluster aliases" aka "user friendly URLs" at the moment (no ETA though), in the meantime, you should get haproxy to inject the header X-Found-Cluster: 708...4e (full id, not the rest of the URL, I shortened for security)

eg

http-request add-header X-Found-Cluster “708...4e”

[balogan (Brett Logan)]

@Alex_Piggott

http-request add-header X-Found-Cluster “708...4e”

Thanks for the quick response.

I made the change and can access through curl on the load balancers, but I'm getting a 503 with a web browser.

< HTTP/1.1 302 Found
< Cache-Control: no-cache
< Content-Length: 0
< Date: Thu, 02 Jan 2020 19:23:50 GMT
< Kbn-Name: kibana
< Kbn-Xpack-Sig: dc7415cd93ad168f271f0f17ac21d92e
< Location: /login?next=%2Fkibana
< X-Cloud-Request-Id: dc4b53ee-80b2-4d35-8397-44dc50bc2740
< X-Found-Handling-Cluster: 708b4ffd5e6f4e8a923f02f0a1f4894e
< X-Found-Handling-Instance: instance-0000000005
< X-Found-Handling-Server: 10.56.56.29
<

• Connection #0 to host vcp-analytics-ecelab.mon.vzwops.com left intact

[Alex_Piggott]

My guess would be that one of the rules in your haproxy is misfiring

Can you get haproxy to log what it's doing exactly (and specifically what it is converting the "input" URL to?)

Or probably if you look in one of the log files in /mnt/data/:id/services/proxy/logs/ it will give some more info

(If you go to the network debugger for the browser and "copy as curl", and then try that curl, I would guess you'll reproduce the error)

[balogan (Brett Logan)]

I got it to work by removing /kibana from the haproxy rules. I could see the initial get going through in the browser network debugger, but the login response is where it failed.

We'll see if my manager is happy removing /kibana. He's on PTO today.

Thanks for the help.

[Rob_wylde]

HAProxy is good and all but below is how i do it with nginx.

upstream ece_proxies {
        zone ece_proxies 64K;
        server 10.1.1.100:9243;
        server 10.1.1.101:9243;
        server 10.1.1.102:9243;
}
#Testing Instance
    server {
            listen 9200 ssl;
            listen 443 ssl; # 'ssl' parameter tells NGINX to decrypt the traffic
            server_name myhouse-es.domain.net;
            ssl_certificate /etc/letsencrypt/live/domain.net/fullchain.pem; # The certificate file
            ssl_certificate_key /etc/letsencrypt/live/domain.net/privkey.pem; # The private key file

        location / {
                proxy_pass https://ece_proxies;
                proxy_http_version 1.1;
                proxy_set_header Host e71625acb09b4befbb40bf03cb3e0d86.domain.net;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Connection "";
        }
}

The above terminated a Lets Encrypted wildcard cert and hands the connections off to the ECE proxies correctly and without and issues.

[Alex_Piggott]

(Setting Host with the full cluster id as a prefix is the alternative to setting X-Found-Cluster, which to use is just a matter of preference)