First of all I'm pretty new to the Kibana world and Haproxy.
Installation :
Centos 7
Haproxy 1.5.18 : installed through yum install haproxy.
Kibana : 6.5 latest release.
It's an 'In the box configuration' with ES, logstash & Redis.
Firewalld stopped.
My problem : Kibana doesn't seem to respond as expected.
Haproxy Configuration :
global
log 127.0.0.1 local1 debug
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
tune.ssl.default-dh-param 2048
# Répertoire SSL par défaut
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
log global
option httplog
timeout connect 5000
timeout client 50000
timeout server 50000
frontend http-in
mode http
bind 0.0.0.0:80
redirect scheme https code 301 if !{ ssl_fc }
frontend https-in
bind 0.0.0.0:443 ssl crt /etc/ssl/private/vmrelkoytst.pem
reqadd X-Forwarded-Proto:\ https
acl acl_kibana path_beg /kibana
use_backend kibana if acl_kibana
backend kibana
mode http
option forwardfor
option httpchk GET /
reqrep ^([^\ :]*)\ /kibana/(.*) \1\ /\2
server relk 127.0.0.1:5601
Haproxy doesn't make a connexion with Kibana was my first impression,
based on the lack of traces in the Kibana log.
When I go to the URL http://myserver/kibana I'm correctly redirected to https://myserver/kibana but I'm having these logs.
haproxy log :
The webpage is giving the error : myserver didn't send any data ( ERR_EMPTY_RESPONSE )
Response to : curl http://x.x.x.x:5601/app/kibana OK
Response to : curl http://x.x.x.x:5601/kibana 404 Error
Response to : curl http://x.x.x.x:5601/ No response
I don't understant why kibana doesn't respond correctly on my 2 last curl's.
Can somebody help me ?
No, It doesn't I suppose because of the 127.0.0.1 param in the kibana config.
Found some information on SELinux blocking ports for HAPROXY, I'm checking this out.
I will keep you posted of course
Of course if you have other ideas
EDIT : Kibana worked before the installation of the HAproxy, if needed I can post my original kibana config ( Information is on another pc.)
Try commenting this out of your Haproxy config file. That is not a proper health check url for Kibana, as evident from your Haproxy log saying "NOSRV" with disconnect status code of "SC" meaning no connection could be made to the backed server. If you looked at the Haproxy stats page with your current setup, you'd probably see your backend server is marked as down/red. No server to connect to...no worky. If you're not using xpack security, the health check url would be /api/status with an expected 200 status code returned (Haproxy is powerful enough you can configure the helath checks to read the returned json looking for some text instead of the 200 code if you so pleased). With xpack security you have several options...Kibana.yml and enable anonymous status access or look for another page. I'd choose the later and look for status 200 on the default login page url.
Well, like I said...if HAProxy is detecting no server available because of a bad health check config, it will never make a connection to Kibana. Setup the HAProxy stats backend to confirm:
Put the following in your HAProxy config file (change IP/port to your correct settings, change username:password to whatever you'd like). Then connect to the page and check the Kibana backend status.
I just took the version that was in the yum repository.
I'll try to install tomorrow the 1.8 version of Haproxy, I'll use this Link
Thanks for your help.
So because the backend server line is grey, it just means that health checking is not enabled (evident in your HAProxy config file that the is no "check" directive on the server line. So...even though you have a junk health check url, it's not being used by HAProxy because you're not telling it to. To enable health checks, the server line would look like this:
If not using an ssl cert on Kibana
server relk 127.0.0.1:5601 check
If using an ssl cert on Kibana
server relk 127.0.0.1:5601 check check-ssl
Instead of having HAProxy rewrite the basepath using your request replace directive (reqrep.....), have you tried removing that from the HAProxy config and then changing the Kibana.yml file option instead: server.rewritebasepath: true
The Haproxy log :
Jan 3 14:22:40 localhost haproxy[11078]: Server kibana/relk is DOWN, reason: Layer7 wrong status, code: 404, info: "Not Found", check duration: 2ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jan 3 14:22:40 localhost haproxy[11078]: backend kibana has no server available!
My conclusion to this :
Haproxy is forwarding the request 'as is' to Kibana which rejects them because they aren't coming from the localhost ( config file says server.host : 127.0.0.1) so he rejects the request and that's why we get the connexion aborted, right ? Can we say to haproxy to send them as the 'localhost' ?
I supposed that the 'basepath' config should allow us to use /kibana instead of /app/kibana, correct ?
You shouldn't have HAProxy rewrite the request path AND also have kibana.yml set to rewritebasepath: true...it's one or the other.
No. If you have your server line in HAProxy config set to 127.0.01:5601, THAT is where the request is forwarded to...127.0.0.1.
Sort of (you can specify a source IP to use), but there should be no need. On the server line, you'd use the "source" directive to tell HAProxy to use a certain IP...in this case, 127.0.0.1. So when a connection is made to a backend server, it comes from 127.0.0.1 instead of the default interface. Again...I don't think you need to do this as I don't believe it's the problem.
Again...I think your problem lies with your basepath config and your'e telling both HAProxy AND Kibana to rewrite the base path...only one should be doing that.
Sorry there has been a mistype in my comment.
The ' reqrep ^([^\ :] )\ /kibana/(. ) \1\ /\2' was commented. I'll change my post immediatly.
sorry for this.
Haproxy is forwarding the request 'as is' to Kibana which rejects them because they aren't coming from the localhost ( config file says server.host : 127.0.0.1) so he rejects the request and that's why we get the connexion aborted, right ?
Haproxy forwards the request ' https://myserver/kibana/app/kibana' as is to kibana and kibana rejects it because of the line in kibana.yml : server.host: 127.0.0.1 which means he accepts only request from the localhost.
Haproxy doesn't change the request ' https://myserver/kibana/app/kibana' to 'https://127.0.0.1/kibana/app/kibana', it's only forwarding it.and that's why the error is 'ERR_CONNECTION_ABORTED'.
That just sets the host address to LISTEN for request on...the name is insignificant. HAProxy is configured properly to send the requests to 127.0.0.1.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
