This finally happened
On a fresh 6.2.2 install everything works fine.
Update, I increased the heap memory and still it tries to optimize and about 30 mins later I get the out of memory errror. Note this is a VM with 8GB max ram and NO data whatsoever.
Could you try to do the same on a Ubuntu VM or something?
8GB is more than sufficient, I have run this on a 4 GB server, and it has worked perfectly.
Hi adwait,
I think I am also facing same issue for Elastic and Kibana 6.5.4
Kibana saying : Kibana Server is not ready yet
Please check below output:
root@ip-10-0-1-26:/home/ubuntu# curl -k -vv -u admin:admin https://localhost:9200/_searchguard/authinfo
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: DC=de; L=test; O=node; OU=node; CN=node-0.example.com
* start date: Apr 22 03:43:47 2018 GMT
* expire date: Apr 19 03:43:47 2028 GMT
* issuer: DC=com; DC=example; O=Example Com Inc.; OU=Example Com Inc. Root CA; CN=Example Com Inc. Root CA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user 'admin'
> GET /_searchguard/authinfo HTTP/1.1
> Host: localhost:9200
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 438
<
* Connection #0 to host localhost left intact
{"user":"User [name=admin, roles=[admin], requestedTenant=null]","user_name":"admin","user_requested_tenant":null,"remote_address":"127.0.0.1:35166","backend_roles":["admin"],"custom_attribute_names":["attr.internal.attribute1","attr.internal.attribute2","attr.internal.attribute3"],"sg_roles":["sg_all_access","sg_own_index"],"sg_tenants":{"admin_tenant":true,"admin":true},"principal":null,"peer_certificates":"0","sso_logout_url":null}root@ip-10-0-1-26:/home/ubuntu#
root@ip-10-0-1-26:/home/ubuntu#
root@ip-10-0-1-26:/home/ubuntu# curl -k -vv -u admin:admin https://localhost:5601/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5601 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=de; L=test; O=client; OU=client; CN=kirk
* start date: Apr 22 03:43:47 2018 GMT
* expire date: Apr 19 03:43:47 2028 GMT
* issuer: DC=com; DC=example; O=Example Com Inc.; OU=Example Com Inc. Root CA; CN=Example Com Inc. Root CA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user 'admin'
> GET / HTTP/1.1
> Host: localhost:5601
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< retry-after: 30
< content-type: text/html; charset=utf-8
< cache-control: no-cache
< content-length: 30
< Date: Wed, 16 Jan 2019 10:17:54 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
Kibana server is not ready yetroot@ip-10-0-1-26:/home/ubuntu#
root@ip-10-0-1-26:/home/ubuntu#
Hello,
Elasticsearch Version: 6.5.4
Kibana Version: 6.5.4
Searchguard Plugin Version: 6.5.4-17
Centos: 7
Running searchguard plugin with Optmize option waits for 20 min and finishes with an error
$ /usr/share/kibana/bin/kibana-plugin install file:///tmp/kitchen/cache/searchguard-6.5.4.zip
(node:1562) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.
Browserslist: caniuse-lite is outdated. Please run next command `npm update caniuse-lite browserslist`
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/usr/share/kibana/node/bin/node]
2: 0x8cce9c [/usr/share/kibana/node/bin/node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/share/kibana/node/bin/node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/share/kibana/node/bin/node]
5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/share/kibana/node/bin/node]
6: v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [/usr/share/kibana/node/bin/node]
7: v8::internal::String::Flatten(v8::internal::Handle<v8::internal::String>, v8::internal::PretenureFlag) [/usr/share/kibana/node/bin/node]
8: v8::internal::String::LastIndexOf(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/share/kibana/node/bin/node]
9: v8::internal::Builtin_StringPrototypeLastIndexOf(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/share/kibana/node/bin/node]
10: 0x129f5948697d
<--- Last few GCs --->
[1562:0x406ba20] 1081414 ms: Mark-sweep 1308.9 (1458.3) -> 1308.8 (1458.8) MB, 1936.6 / 0.0 ms allocation failure GC in old space requested
[1562:0x406ba20] 1083236 ms: Mark-sweep 1308.8 (1458.8) -> 1308.8 (1427.3) MB, 1822.1 / 0.0 ms last resort GC in old space requested
[1562:0x406ba20] 1084934 ms: Mark-sweep 1308.8 (1427.3) -> 1308.8 (1427.3) MB, 1697.4 / 0.0 ms last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x1e123d258b9 <JSObject>
0: builtin exit frame: lastIndexOf(this=0x27e65b5f18e1 <Very long string[1106657]>,0x237a12120e9 <String[1]\: \n>)
1: has_nlb(aka has_nlb) [0x155e292022d1 <undefined>:5970] [bytecode=0x1c34d018bc41 offset=15](this=0x155e292022d1 <undefined>)
2: /* anonymous */(aka /* anonymous */) [0x155e292022d1 <undefined>:6070] [bytecode=0x32aee639c8d1 offset=60](this=0x155e292022d1 <undefined>,c=...
[root@kibana-centos-7 bin]# /usr/share/kibana/node/bin/node /usr/share/kibana/src/cli --env.name=production --optimize.useBundleCache=false --server.autoListen=false --plugins.initialize=false
(node:2773) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.
Browserslist: caniuse-lite is outdated. Please run next command `npm update caniuse-lite browserslist`
Follows in the next post...
Further running kibana in a verbose mode gave me few errors on getConnections. I guess these are memory errors.
/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml -l stdout
log [17:01:23.875] [debug][config] Marking config path as handled: server
log [17:01:23.875] [debug][http] registering route handler for [/core]
log [17:01:23.876] [debug][root] starting root
log [17:01:23.880] [debug][config] Marking config path as handled: logging
log [17:01:24.001] [debug][server] starting server
log [17:01:24.004] [debug][server][http] starting http server
log [17:01:24.036] [debug][server][http] http server running at https://0.0.0.0:5601
log [17:01:24.037] [debug][legacy][service] starting legacy service
log [17:01:25.194] [debug][config] Marking config path as handled: server
log [17:01:25.390] [debug][pid] wrote pid file to /var/run/kibana/kibana.pid
log [17:01:40.834] [debug][plugin] Found plugin at /usr/share/kibana/node_modules/x-pack
log [17:01:40.841] [debug][legacy][proxy] "getConnections" has been called.
log [17:01:40.845] [debug][legacy][proxy] Event is being forwarded: connection
ops [17:01:40.845] memory: 194.4MB uptime: 0:00:18 load: [4.40 4.02 3.29] delay: 4.434
log [17:01:41.004] [debug][plugin] Found plugin at /usr/share/kibana/plugins/searchguard
log [17:01:41.006] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/apm_oss
log [17:01:41.014] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/console
log [17:01:41.022] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/elasticsearch
log [17:01:41.024] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/input_control_vis
log [17:01:41.025] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/inspector_views
log [17:01:41.026] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/kbn_doc_views
log [17:01:41.028] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/kbn_vislib_vis_types
log [17:01:41.095] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/kibana
log [17:01:41.102] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/markdown_vis
log [17:01:41.103] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/metric_vis
log [17:01:41.151] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/metrics
log [17:01:41.153] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/region_map
log [17:01:41.155] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/state_session_storage_redirect
log [17:01:41.156] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/status_page
log [17:01:41.166] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/table_vis
log [17:01:41.167] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/tagcloud
log [17:01:41.168] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/tile_map
log [17:01:41.170] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/timelion
log [17:01:41.172] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/vega
log [17:01:41.699] [debug][legacy][service] Kibana server is not ready yet GET:/.
log [17:01:42.093] [info][optimize] Optimizing and caching bundles for ml, stateSessionStorageRedirect, status_page, timelion, graph, monitoring, space_selector, login, logout, dashboardViewer, apm, canvas, infra, searchguard-login, searchguard-customerror, searchguard-multitenancy, searchguard-accountinfo, searchguard-configuration and kibana. This may take a few minutes
log [17:01:45.869] [debug][legacy][proxy] "getConnections" has been called.
ops [17:01:45.924] memory: 174.9MB uptime: 0:00:23 load: [4.37 4.02 3.30] delay: 54.847
Browserslist: caniuse-lite is outdated. Please run next command `npm update caniuse-lite browserslist`
log [17:01:50.920] [debug][legacy][proxy] "getConnections" has been called.
ops [17:01:50.953] memory: 252.8MB uptime: 0:00:28 load: [4.50 4.05 3.31] delay: 33.536
log [17:01:55.939] [debug][legacy][proxy] "getConnections" has been called.
ops [17:01:56.092] memory: 260.6MB uptime: 0:00:33 load: [4.46 4.05 3.32] delay: 153.010
log [17:02:00.948] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:00.953] memory: 228.3MB uptime: 0:00:38 load: [4.26 4.01 3.31] delay: 4.598
log [17:02:06.262] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:06.278] memory: 251.8MB uptime: 0:00:44 load: [4.24 4.01 3.31] delay: 16.178
log [17:02:11.312] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:11.491] memory: 228.9MB uptime: 0:00:49 load: [4.46 4.06 3.33] delay: 178.608
log [17:02:16.396] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:16.430] memory: 336.9MB uptime: 0:00:54 load: [4.74 4.13 3.36] delay: 34.560
log [17:02:21.489] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:21.589] memory: 280.3MB uptime: 0:00:59 load: [5.00 4.19 3.38] delay: 100.217
log [17:02:27.020] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:27.197] memory: 241.0MB uptime: 0:01:04 load: [4.76 4.16 3.38] delay: 177.221
log [17:02:32.214] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:32.433] memory: 390.3MB uptime: 0:01:10 load: [4.78 4.17 3.38] delay: 218.692
log [17:02:37.217] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:37.550] memory: 317.9MB uptime: 0:01:15 load: [4.80 4.19 3.39] delay: 333.571
Also curl response for localhost:5601
curl https://localhost:5601 -vk
* About to connect() to localhost port 5601 (#0)
* Trying ::1...
* Connection refused
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5601 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=node.elastictest.com,OU=Ops,O="elastictest Com, Inc.",DC=elastictest,DC=com
* start date: May 18 19:50:26 2018 GMT
* expire date: May 15 19:50:26 2028 GMT
* common name: node.elastictest.com
* issuer: CN=signing.ca.elastictest.com,OU=CA,O="elastictest Com, Inc.",DC=elastictest,DC=com
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:5601
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< retry-after: 30
< content-type: text/html; charset=utf-8
< cache-control: no-cache
< content-length: 30
< Date: Mon, 21 Jan 2019 17:01:41 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
Kibana server is not ready yet
Thanks
Dinesh
Is there any update to this from the Elastic staff? Is this an issue with kibana optimization process?