Elastic Search Behind Apache Reverse Proxy But Kibana Faces Problems Accessing ES

Hello Everyone,

This is my configuration -

[ Elastic Search 1.6 :9200] <--- [Apache Web Server 2.4 Reverse Proxy :443/es ] <-- [Kibana 4.1 HTTPS and Port 5601]

Both Apache Reverse Proxy and Kibana are setup to use HTTPS with their own server side certificates signed by the same internal CA.

Now, when I hit the Apache URL, e.g. https://es.my.web.server.com:443/es/ , I receive a response similar to the following, which tells me everything is fine behind the reverse proxy -

{
    "status" : 200,
    "name" : "Sgsi.Default.Es.Node",
    "cluster_name" : "sgsi_es_local",
    "version" : {
        "number" : "1.6.0",
        "build_hash" : "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0",
        "build_timestamp" : "2015-06-09T13:36:34Z",
        "build_snapshot" : false,
        "lucene_version" : "4.10.4"
    },
    "tagline" : "You Know, for Search"
}

I tried other commands as well, like _alias, _cat, _search etc. all seems to work fine.

However, when I pointed Kibana to the Reverse Proxy by updating the Kibana.yml (see below), everything was fine when I started Kibana until I hit Kibana with a browser -

port: 5601
host: "0.0.0.0"
elasticsearch_preserve_host: false
kibana_index: ".kibana"
default_app_id: "discover"
request_timeout: 300000
shard_timeout: 0

# The Elasticsearch instance to use for all your queries.
elasticsearch_url: "https://es.my.web.server.com:443/es/"

# If your Elasticsearch is protected with basic auth, this is the user
# credentials used by the Kibana server to perform maintence on the 
# kibana_index at statup. Your Kibana users will still need to authenticate
# with Elasticsearch (which is proxied thorugh the Kibana server)
kibana_elasticsearch_username: esu
kibana_elasticsearch_password: esu123

# If your Elasticsearch requires client certificate and key
# kibana_elasticsearch_client_crt: /path/to/your/client.crt
# kibana_elasticsearch_client_key: /path/to/your/client.key

# If you need to provide a CA certificate for your Elasticsarech instance,
# put the path of the pem file here.
ca: /Users/vpathak/Software/kibana/config/ca_cert.crt

# Set to false to have a complete disregard for the validity of the SSL
# certificate.
verify_ssl: true

# SSL for outgoing requests from the Kibana Server (PEM formatted)
ssl_key_file: /Users/vpathak/Software/kibana/config/kbn_private.key
ssl_cert_file: /Users/vpathak/Software/kibana/config/kbn.crt

When I try to access Kibana by visiting https://sgsi.kbn.local.com:5601, Kibana appear to load for 2-3 seconds and then error out as this -

Oooops

< Go Back >      < Clear Your Session >

Error: Bad Gateway
at respond (https://sgsi.kbn.local.com:5601/index.js?_b=7562:85289:15)
at checkRespForFailure (https://sgsi.kbn.local.com:5601/index.js?_b=7562:85257:7)
at https://sgsi.kbn.local.com:5601/index.js?_b=7562:83895:7
at wrappedErrback (https://sgsi.kbn.local.com:5601/index.js?_b=7562:20902:78)
at https://sgsi.kbn.local.com:5601/index.js?_b=7562:21035:76
at Scope.$apply (https://sgsi.kbn.local.com:5601/index.js?_b=7562:22126:24)

When I comment ssl_key_file and ssl_cert_file and try to access Kibana over HTTP, the result is same that it still errors out, but the error is different this time -

Oooops

< Go Back >      < Clear Your Session >

Error: Unknown error while connecting to Elasticsearch
  ErrorAbstract@http://sgsi.es.local.com:5601/index.js?_b=7562:83947:19
  StatusCodeError@http://sgsi.es.local.com:5601/index.js?_b=7562:84096:5
  respond@http://sgsi.es.local.com:5601/index.js?_b=7562:85289:15
  checkRespForFailure@http://sgsi.es.local.com:5601/index.js?_b=7562:85257:7
  [24]</AngularConnector.prototype.request/<@http://sgsi.es.local.com:5601/index.js?_b=7562:83895:7
  qFactory/createInternalRejectedPromise/<.then/<@http://sgsi.es.local.com:5601/index.js?_b=7562:21035:29
  $RootScopeProvider/this.$get</Scope.prototype.$eval@http://sgsi.es.local.com:5601/index.js?_b=7562:22022:16
  $RootScopeProvider/this.$get</Scope.prototype.$digest@http://sgsi.es.local.com:5601/index.js?_b=7562:21834:15
  $RootScopeProvider/this.$get</Scope.prototype.$apply@http://sgsi.es.local.com:5601/index.js?_b=7562:22126:13
  done@http://sgsi.es.local.com:5601/index.js?_b=7562:17661:34
  completeRequest@http://sgsi.es.local.com:5601/index.js?_b=7562:17875:7
  createHttpBackend/</xhr.onreadystatechange@http://sgsi.es.local.com:5601/index.js?_b=7562:17814:1

I have no clue What is the problem Kibana is facing and what Configuration change can I do to make Kibana comfortable.

Can someone see anything wrong here ?

Thanks,

(*Vipul)() ;

Can you post your full kibana.yml config?

Thanks Tanya for the Response. Apologies that I was not able to reply earlier.

Here is my full Kibana.yml configuration -

# Kibana is served by a back end server. This controls which port to use.
port: 5601
host: "0.0.0.0"
elasticsearch_url: "http://sgsi.es.local.com/el"

# preserve_elasticsearch_host true will send the hostname specified in elasticsearch.
elasticsearch_preserve_host: true

# Kibana uses an index in Elasticsearch to store saved searches.
kibana_index: ".kibana"

# If your Elasticsearch is protected with basic auth, this is the user
# credentials used by the Kibana server to perform maintence on the 
# kibana_index at statup. Your Kibana users will still need to authenticate
# with Elasticsearch (which is proxied thorugh the Kibana server)
kibana_elasticsearch_username: esu
kibana_elasticsearch_password: esu123

# If your Elasticsearch requires client certificate and key
# kibana_elasticsearch_client_crt: /path/to/your/client.crt
# kibana_elasticsearch_client_key: /path/to/your/client.key

# If you need to provide a CA certificate for your Elasticsarech instance
ca: /Users/vpathak/Software/kibana/config/ca_cert.crt

# The default application to load.
default_app_id: "discover"

# ping_timeout: 1500
request_timeout: 300000
shard_timeout: 0
# startup_timeout: 5000

# Set to false to have a complete disregard for the validity of the SSL
# certificate.
verify_ssl: true

# SSL for outgoing requests from the Kibana Server (PEM formatted)
ssl_key_file: /Users/vpathak/Software/kibana/config/kbn_private.key
ssl_cert_file: /Users/vpathak/Software/kibana/config/kbn.crt

# Set the path to where you would like the process id file to be created.
pid_file: /var/run/kibana.pid
# log_file: ./kibana.log

# Plugins that are included in the build, and no longer found in the plugins/ folder
bundled_plugin_ids:
 - plugins/dashboard/index
 - plugins/discover/index
 - plugins/doc/index
 - plugins/kibana/index
 - plugins/markdown_vis/index
 - plugins/metric_vis/index
 - plugins/settings/index
 - plugins/table_vis/index
 - plugins/vis_types/index
 - plugins/visualize/index

Additionally, here is my Reverse Proxy configuration -

# Rewrite URL that starts and ends with /es. Redirect it to /es/
RewriteEngine On
RewriteRule "^/es/es/" "/es/" [R] 
RewriteRule "^/es$" "/es/" [R] 

# Set Standard (Forward) Proxy Off ... 
ProxyRequests off 
ProxyPreserveHost On

<Location /es/ >
  Options All 
  Order deny,allow
  Allow from all 

  # Define the actual proxy mapping and redirection
  LogLevel debug
  ProxyPass                   http://localhost:9200/ retry=0 timeout=5
  ProxyPassReverse            http://localhost:9200/
  ProxyPassReverseCookiePath  / /es/
  ProxyHTMLURLMap             / /es/
</Location>

The Apache access logs are showing evidence that Kibana is trying to access /es/es URL, but I have no idea why and how, a request could be sent to /es/es when I clearly configured Elastic Search to http://localhost:9200/es -

127.0.0.1 - - [08/Oct/2015:15:39:25 -0700] "HEAD /es/ HTTP/1.1" 200 -
127.0.0.1 - - [08/Oct/2015:15:39:25 -0700] "POST /es/es/_mget?timeout=0&ignore_unavailable=true&preference=1444343965639 HTTP/1.1" 302 279
127.0.0.1 - - [08/Oct/2015:15:39:25 -0700] "HEAD /es/es/ HTTP/1.1" 302 -
127.0.0.1 - - [08/Oct/2015:15:39:25 -0700] "HEAD /es/ HTTP/1.1" 200 -
127.0.0.1 - - [08/Oct/2015:15:39:25 -0700] "POST /es/es/_mget?timeout=0&ignore_unavailable=true&preference=1444343965639 HTTP/1.1" 302 279
127.0.0.1 - - [08/Oct/2015:15:39:25 -0700] "HEAD /es/es/ HTTP/1.1" 302 -
127.0.0.1 - - [08/Oct/2015:15:39:25 -0700] "HEAD /es/ HTTP/1.1" 200 -
127.0.0.1 - - [08/Oct/2015:15:39:25 -0700] "POST /es/es/_mget?timeout=0&ignore_unavailable=true&preference=1444343965639 HTTP/1.1" 302 279
127.0.0.1 - - [08/Oct/2015:15:39:25 -0700] "HEAD /es/es/ HTTP/1.1" 302 -
127.0.0.1 - - [08/Oct/2015:15:39:25 -0700] "HEAD /es/ HTTP/1.1" 200 -
127.0.0.1 - - [08/Oct/2015:15:39:25 -0700] "POST /es/es/_mget?timeout=0&ignore_unavailable=true&preference=1444343965639 HTTP/1.1" 302 279

I even added a rewrite Rule in the proxy configuration of convert /es/es to a single /es/, but there is no effect.

Thanks for your help and efforts.

Regards,

(*Vipul)() ;

I know this is a year old thread.. but I ended up here trying to find rules for getting elasticsearch and kibana to be virtuals behind apache. The config here helped, and I was able to figure it out ( so far anyway ). You're on the right track, but you have to alias it to elasticsearch, not es, for kibana to work correctly. There are calls out to an elasticsearch url in the logs. Not sure why yet, probably some n00b error on my part with a setting. After that, I had to create aliases for app ( for kibana and sense ) and bundles ( for the url kibana uses to get its guts ). Here's what I ended up with ( ProxyHTMLURLMap didn't work on my apache install.. so I left it of. Still works ).
I'd be happy to hear I'm still doing this wrong, and it's all easier than this. Not secure now, but I'm just trying to get it up and running to test it out.

Rewrite URL that starts and ends with /elasticsearch. Redirect it to /elasticsearch/

RewriteEngine On
RewriteRule "^/elasticsearch/elasticsearch/" "/elasticsearch/" [R]
RewriteRule "^/elasticsearch$" "/elasticssearch/" [R]

RewriteRule "^/app/kibana/" "/app/" [R]
RewriteRule "^/app$" "/app/" [R]

RewriteRule "^/bundles/bundles/" "/bundles/" [R]
RewriteRule "^/bundles$" "/bundles/" [R]

Set Standard (Forward) Proxy Off ...

ProxyRequests off
ProxyPreserveHost On

<Location /elasticsearch/ >
Options All
Order deny,allow
Allow from all

Define the actual proxy mapping and redirection

LogLevel debug
ProxyPass http://localhost:9200/ retry=0 timeout=5
ProxyPassReverse http://localhost:9200/
ProxyPassReverseCookiePath / /elasticsearch/

<Location /app/ >
Options All
Order deny,allow
Allow from all

LogLevel debug
ProxyPass http://localhost:5601/app/ retry=0 timeout=5
ProxyPassReverse http://localhost:5601/app/
ProxyPassReverseCookiePath / /app/

<Location /bundles/ >
Options All
Order deny,allow
Allow from all

LogLevel debug
ProxyPass http://localhost:5601/bundles/ retry=0 timeout=5
ProxyPassReverse http://localhost:5601/bundles/
ProxyPassReverseCookiePath / /app/