Kibana Server Not Ready

This finally happened

image

On a fresh 6.2.2 install everything works fine.

Update, I increased the heap memory and still it tries to optimize and about 30 mins later I get the out of memory errror. Note this is a VM with 8GB max ram and NO data whatsoever.

Could you try to do the same on a Ubuntu VM or something?

8GB is more than sufficient, I have run this on a 4 GB server, and it has worked perfectly.

Hi adwait,

I think I am also facing same issue for Elastic and Kibana 6.5.4

Kibana saying : Kibana Server is not ready yet

Please check below output:

root@ip-10-0-1-26:/home/ubuntu# curl -k -vv -u admin:admin https://localhost:9200/_searchguard/authinfo
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: DC=de; L=test; O=node; OU=node; CN=node-0.example.com
*  start date: Apr 22 03:43:47 2018 GMT
*  expire date: Apr 19 03:43:47 2028 GMT
*  issuer: DC=com; DC=example; O=Example Com Inc.; OU=Example Com Inc. Root CA; CN=Example Com Inc. Root CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user 'admin'
> GET /_searchguard/authinfo HTTP/1.1
> Host: localhost:9200
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 438
< 
* Connection #0 to host localhost left intact
{"user":"User [name=admin, roles=[admin], requestedTenant=null]","user_name":"admin","user_requested_tenant":null,"remote_address":"127.0.0.1:35166","backend_roles":["admin"],"custom_attribute_names":["attr.internal.attribute1","attr.internal.attribute2","attr.internal.attribute3"],"sg_roles":["sg_all_access","sg_own_index"],"sg_tenants":{"admin_tenant":true,"admin":true},"principal":null,"peer_certificates":"0","sso_logout_url":null}root@ip-10-0-1-26:/home/ubuntu# 
root@ip-10-0-1-26:/home/ubuntu# 
root@ip-10-0-1-26:/home/ubuntu# curl -k -vv -u admin:admin https://localhost:5601/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5601 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=de; L=test; O=client; OU=client; CN=kirk
*  start date: Apr 22 03:43:47 2018 GMT
*  expire date: Apr 19 03:43:47 2028 GMT
*  issuer: DC=com; DC=example; O=Example Com Inc.; OU=Example Com Inc. Root CA; CN=Example Com Inc. Root CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user 'admin'
> GET / HTTP/1.1
> Host: localhost:5601
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 503 Service Unavailable
< retry-after: 30
< content-type: text/html; charset=utf-8
< cache-control: no-cache
< content-length: 30
< Date: Wed, 16 Jan 2019 10:17:54 GMT
< Connection: keep-alive
< 
* Connection #0 to host localhost left intact
Kibana server is not ready yetroot@ip-10-0-1-26:/home/ubuntu# 
root@ip-10-0-1-26:/home/ubuntu# 

Hello,

Elasticsearch Version: 6.5.4
Kibana Version: 6.5.4
Searchguard Plugin Version: 6.5.4-17
Centos: 7

Running searchguard plugin with Optmize option waits for 20 min and finishes with an error

$ /usr/share/kibana/bin/kibana-plugin install file:///tmp/kitchen/cache/searchguard-6.5.4.zip
(node:1562) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.
Browserslist: caniuse-lite is outdated. Please run next command `npm update caniuse-lite browserslist`
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/share/kibana/node/bin/node]
 2: 0x8cce9c [/usr/share/kibana/node/bin/node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/share/kibana/node/bin/node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/share/kibana/node/bin/node]
 5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/share/kibana/node/bin/node]
 6: v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [/usr/share/kibana/node/bin/node]
 7: v8::internal::String::Flatten(v8::internal::Handle<v8::internal::String>, v8::internal::PretenureFlag) [/usr/share/kibana/node/bin/node]
 8: v8::internal::String::LastIndexOf(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/share/kibana/node/bin/node]
 9: v8::internal::Builtin_StringPrototypeLastIndexOf(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/share/kibana/node/bin/node]
10: 0x129f5948697d
<--- Last few GCs --->
[1562:0x406ba20]  1081414 ms: Mark-sweep 1308.9 (1458.3) -> 1308.8 (1458.8) MB, 1936.6 / 0.0 ms  allocation failure GC in old space requested
[1562:0x406ba20]  1083236 ms: Mark-sweep 1308.8 (1458.8) -> 1308.8 (1427.3) MB, 1822.1 / 0.0 ms  last resort GC in old space requested
[1562:0x406ba20]  1084934 ms: Mark-sweep 1308.8 (1427.3) -> 1308.8 (1427.3) MB, 1697.4 / 0.0 ms  last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x1e123d258b9 <JSObject>
    0: builtin exit frame: lastIndexOf(this=0x27e65b5f18e1 <Very long string[1106657]>,0x237a12120e9 <String[1]\: \n>)

    1: has_nlb(aka has_nlb) [0x155e292022d1 <undefined>:5970] [bytecode=0x1c34d018bc41 offset=15](this=0x155e292022d1 <undefined>)
    2: /* anonymous */(aka /* anonymous */) [0x155e292022d1 <undefined>:6070] [bytecode=0x32aee639c8d1 offset=60](this=0x155e292022d1 <undefined>,c=...
[root@kibana-centos-7 bin]#  /usr/share/kibana/node/bin/node /usr/share/kibana/src/cli --env.name=production --optimize.useBundleCache=false --server.autoListen=false --plugins.initialize=false
(node:2773) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.
Browserslist: caniuse-lite is outdated. Please run next command `npm update caniuse-lite browserslist`

Follows in the next post...

Further running kibana in a verbose mode gave me few errors on getConnections. I guess these are memory errors.

/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml -l stdout
  log   [17:01:23.875] [debug][config] Marking config path as handled: server
  log   [17:01:23.875] [debug][http] registering route handler for [/core]
  log   [17:01:23.876] [debug][root] starting root
  log   [17:01:23.880] [debug][config] Marking config path as handled: logging
  log   [17:01:24.001] [debug][server] starting server
  log   [17:01:24.004] [debug][server][http] starting http server
  log   [17:01:24.036] [debug][server][http] http server running at https://0.0.0.0:5601
  log   [17:01:24.037] [debug][legacy][service] starting legacy service
  log   [17:01:25.194] [debug][config] Marking config path as handled: server
  log   [17:01:25.390] [debug][pid] wrote pid file to /var/run/kibana/kibana.pid
  log   [17:01:40.834] [debug][plugin] Found plugin at /usr/share/kibana/node_modules/x-pack
  log   [17:01:40.841] [debug][legacy][proxy] "getConnections" has been called.
  log   [17:01:40.845] [debug][legacy][proxy] Event is being forwarded: connection
  ops   [17:01:40.845]  memory: 194.4MB uptime: 0:00:18 load: [4.40 4.02 3.29] delay: 4.434
  log   [17:01:41.004] [debug][plugin] Found plugin at /usr/share/kibana/plugins/searchguard
  log   [17:01:41.006] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/apm_oss
  log   [17:01:41.014] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/console
  log   [17:01:41.022] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/elasticsearch
  log   [17:01:41.024] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/input_control_vis
  log   [17:01:41.025] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/inspector_views
  log   [17:01:41.026] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/kbn_doc_views
  log   [17:01:41.028] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/kbn_vislib_vis_types
  log   [17:01:41.095] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/kibana
  log   [17:01:41.102] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/markdown_vis
  log   [17:01:41.103] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/metric_vis
  log   [17:01:41.151] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/metrics
  log   [17:01:41.153] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/region_map
  log   [17:01:41.155] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/state_session_storage_redirect
  log   [17:01:41.156] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/status_page
  log   [17:01:41.166] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/table_vis
  log   [17:01:41.167] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/tagcloud
  log   [17:01:41.168] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/tile_map
  log   [17:01:41.170] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/timelion
  log   [17:01:41.172] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/vega
  log   [17:01:41.699] [debug][legacy][service] Kibana server is not ready yet GET:/.
  log   [17:01:42.093] [info][optimize] Optimizing and caching bundles for ml, stateSessionStorageRedirect, status_page, timelion, graph, monitoring, space_selector, login, logout, dashboardViewer, apm, canvas, infra, searchguard-login, searchguard-customerror, searchguard-multitenancy, searchguard-accountinfo, searchguard-configuration and kibana. This may take a few minutes
  log   [17:01:45.869] [debug][legacy][proxy] "getConnections" has been called.
  ops   [17:01:45.924]  memory: 174.9MB uptime: 0:00:23 load: [4.37 4.02 3.30] delay: 54.847
Browserslist: caniuse-lite is outdated. Please run next command `npm update caniuse-lite browserslist`
  log   [17:01:50.920] [debug][legacy][proxy] "getConnections" has been called.
  ops   [17:01:50.953]  memory: 252.8MB uptime: 0:00:28 load: [4.50 4.05 3.31] delay: 33.536
  log   [17:01:55.939] [debug][legacy][proxy] "getConnections" has been called.
  ops   [17:01:56.092]  memory: 260.6MB uptime: 0:00:33 load: [4.46 4.05 3.32] delay: 153.010
  log   [17:02:00.948] [debug][legacy][proxy] "getConnections" has been called.
  ops   [17:02:00.953]  memory: 228.3MB uptime: 0:00:38 load: [4.26 4.01 3.31] delay: 4.598
  log   [17:02:06.262] [debug][legacy][proxy] "getConnections" has been called.
  ops   [17:02:06.278]  memory: 251.8MB uptime: 0:00:44 load: [4.24 4.01 3.31] delay: 16.178
  log   [17:02:11.312] [debug][legacy][proxy] "getConnections" has been called.
  ops   [17:02:11.491]  memory: 228.9MB uptime: 0:00:49 load: [4.46 4.06 3.33] delay: 178.608
  log   [17:02:16.396] [debug][legacy][proxy] "getConnections" has been called.
  ops   [17:02:16.430]  memory: 336.9MB uptime: 0:00:54 load: [4.74 4.13 3.36] delay: 34.560
  log   [17:02:21.489] [debug][legacy][proxy] "getConnections" has been called.
  ops   [17:02:21.589]  memory: 280.3MB uptime: 0:00:59 load: [5.00 4.19 3.38] delay: 100.217
  log   [17:02:27.020] [debug][legacy][proxy] "getConnections" has been called.
  ops   [17:02:27.197]  memory: 241.0MB uptime: 0:01:04 load: [4.76 4.16 3.38] delay: 177.221
  log   [17:02:32.214] [debug][legacy][proxy] "getConnections" has been called.
  ops   [17:02:32.433]  memory: 390.3MB uptime: 0:01:10 load: [4.78 4.17 3.38] delay: 218.692
  log   [17:02:37.217] [debug][legacy][proxy] "getConnections" has been called.
  ops   [17:02:37.550]  memory: 317.9MB uptime: 0:01:15 load: [4.80 4.19 3.39] delay: 333.571

Also curl response for localhost:5601

curl https://localhost:5601 -vk
* About to connect() to localhost port 5601 (#0)
*   Trying ::1...
* Connection refused
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5601 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=node.elastictest.com,OU=Ops,O="elastictest Com, Inc.",DC=elastictest,DC=com
* 	start date: May 18 19:50:26 2018 GMT
* 	expire date: May 15 19:50:26 2028 GMT
* 	common name: node.elastictest.com
* 	issuer: CN=signing.ca.elastictest.com,OU=CA,O="elastictest Com, Inc.",DC=elastictest,DC=com
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:5601
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< retry-after: 30
< content-type: text/html; charset=utf-8
< cache-control: no-cache
< content-length: 30
< Date: Mon, 21 Jan 2019 17:01:41 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
Kibana server is not ready yet

Thanks
Dinesh

Is there any update to this from the Elastic staff? Is this an issue with kibana optimization process?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.