This finally happened
On a fresh 6.2.2 install everything works fine.
Update, I increased the heap memory and still it tries to optimize and about 30 mins later I get the out of memory errror. Note this is a VM with 8GB max ram and NO data whatsoever.
Could you try to do the same on a Ubuntu VM or something?
8GB is more than sufficient, I have run this on a 4 GB server, and it has worked perfectly.
Hi adwait,
I think I am also facing same issue for Elastic and Kibana 6.5.4
Kibana saying : Kibana Server is not ready yet
Please check below output:
root@ip-10-0-1-26:/home/ubuntu# curl -k -vv -u admin:admin https://localhost:9200/_searchguard/authinfo
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: DC=de; L=test; O=node; OU=node; CN=node-0.example.com
* start date: Apr 22 03:43:47 2018 GMT
* expire date: Apr 19 03:43:47 2028 GMT
* issuer: DC=com; DC=example; O=Example Com Inc.; OU=Example Com Inc. Root CA; CN=Example Com Inc. Root CA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user 'admin'
> GET /_searchguard/authinfo HTTP/1.1
> Host: localhost:9200
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 438
<
* Connection #0 to host localhost left intact
{"user":"User [name=admin, roles=[admin], requestedTenant=null]","user_name":"admin","user_requested_tenant":null,"remote_address":"127.0.0.1:35166","backend_roles":["admin"],"custom_attribute_names":["attr.internal.attribute1","attr.internal.attribute2","attr.internal.attribute3"],"sg_roles":["sg_all_access","sg_own_index"],"sg_tenants":{"admin_tenant":true,"admin":true},"principal":null,"peer_certificates":"0","sso_logout_url":null}root@ip-10-0-1-26:/home/ubuntu#
root@ip-10-0-1-26:/home/ubuntu#
root@ip-10-0-1-26:/home/ubuntu# curl -k -vv -u admin:admin https://localhost:5601/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5601 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=de; L=test; O=client; OU=client; CN=kirk
* start date: Apr 22 03:43:47 2018 GMT
* expire date: Apr 19 03:43:47 2028 GMT
* issuer: DC=com; DC=example; O=Example Com Inc.; OU=Example Com Inc. Root CA; CN=Example Com Inc. Root CA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user 'admin'
> GET / HTTP/1.1
> Host: localhost:5601
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< retry-after: 30
< content-type: text/html; charset=utf-8
< cache-control: no-cache
< content-length: 30
< Date: Wed, 16 Jan 2019 10:17:54 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
Kibana server is not ready yetroot@ip-10-0-1-26:/home/ubuntu#
root@ip-10-0-1-26:/home/ubuntu#
Hello,
Elasticsearch Version: 6.5.4
Kibana Version: 6.5.4
Searchguard Plugin Version: 6.5.4-17
Centos: 7
Running searchguard plugin with Optmize option waits for 20 min and finishes with an error
$ /usr/share/kibana/bin/kibana-plugin install file:///tmp/kitchen/cache/searchguard-6.5.4.zip
(node:1562) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.
Browserslist: caniuse-lite is outdated. Please run next command `npm update caniuse-lite browserslist`
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/usr/share/kibana/node/bin/node]
2: 0x8cce9c [/usr/share/kibana/node/bin/node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/share/kibana/node/bin/node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/share/kibana/node/bin/node]
5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/share/kibana/node/bin/node]
6: v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [/usr/share/kibana/node/bin/node]
7: v8::internal::String::Flatten(v8::internal::Handle<v8::internal::String>, v8::internal::PretenureFlag) [/usr/share/kibana/node/bin/node]
8: v8::internal::String::LastIndexOf(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/share/kibana/node/bin/node]
9: v8::internal::Builtin_StringPrototypeLastIndexOf(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/share/kibana/node/bin/node]
10: 0x129f5948697d
<--- Last few GCs --->
[1562:0x406ba20] 1081414 ms: Mark-sweep 1308.9 (1458.3) -> 1308.8 (1458.8) MB, 1936.6 / 0.0 ms allocation failure GC in old space requested
[1562:0x406ba20] 1083236 ms: Mark-sweep 1308.8 (1458.8) -> 1308.8 (1427.3) MB, 1822.1 / 0.0 ms last resort GC in old space requested
[1562:0x406ba20] 1084934 ms: Mark-sweep 1308.8 (1427.3) -> 1308.8 (1427.3) MB, 1697.4 / 0.0 ms last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x1e123d258b9 <JSObject>
0: builtin exit frame: lastIndexOf(this=0x27e65b5f18e1 <Very long string[1106657]>,0x237a12120e9 <String[1]\: \n>)
1: has_nlb(aka has_nlb) [0x155e292022d1 <undefined>:5970] [bytecode=0x1c34d018bc41 offset=15](this=0x155e292022d1 <undefined>)
2: /* anonymous */(aka /* anonymous */) [0x155e292022d1 <undefined>:6070] [bytecode=0x32aee639c8d1 offset=60](this=0x155e292022d1 <undefined>,c=...
[root@kibana-centos-7 bin]# /usr/share/kibana/node/bin/node /usr/share/kibana/src/cli --env.name=production --optimize.useBundleCache=false --server.autoListen=false --plugins.initialize=false
(node:2773) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.
Browserslist: caniuse-lite is outdated. Please run next command `npm update caniuse-lite browserslist`
Follows in the next post...
Further running kibana in a verbose mode gave me few errors on getConnections. I guess these are memory errors.
/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml -l stdout
log [17:01:23.875] [debug][config] Marking config path as handled: server
log [17:01:23.875] [debug][http] registering route handler for [/core]
log [17:01:23.876] [debug][root] starting root
log [17:01:23.880] [debug][config] Marking config path as handled: logging
log [17:01:24.001] [debug][server] starting server
log [17:01:24.004] [debug][server][http] starting http server
log [17:01:24.036] [debug][server][http] http server running at https://0.0.0.0:5601
log [17:01:24.037] [debug][legacy][service] starting legacy service
log [17:01:25.194] [debug][config] Marking config path as handled: server
log [17:01:25.390] [debug][pid] wrote pid file to /var/run/kibana/kibana.pid
log [17:01:40.834] [debug][plugin] Found plugin at /usr/share/kibana/node_modules/x-pack
log [17:01:40.841] [debug][legacy][proxy] "getConnections" has been called.
log [17:01:40.845] [debug][legacy][proxy] Event is being forwarded: connection
ops [17:01:40.845] memory: 194.4MB uptime: 0:00:18 load: [4.40 4.02 3.29] delay: 4.434
log [17:01:41.004] [debug][plugin] Found plugin at /usr/share/kibana/plugins/searchguard
log [17:01:41.006] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/apm_oss
log [17:01:41.014] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/console
log [17:01:41.022] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/elasticsearch
log [17:01:41.024] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/input_control_vis
log [17:01:41.025] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/inspector_views
log [17:01:41.026] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/kbn_doc_views
log [17:01:41.028] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/kbn_vislib_vis_types
log [17:01:41.095] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/kibana
log [17:01:41.102] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/markdown_vis
log [17:01:41.103] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/metric_vis
log [17:01:41.151] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/metrics
log [17:01:41.153] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/region_map
log [17:01:41.155] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/state_session_storage_redirect
log [17:01:41.156] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/status_page
log [17:01:41.166] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/table_vis
log [17:01:41.167] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/tagcloud
log [17:01:41.168] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/tile_map
log [17:01:41.170] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/timelion
log [17:01:41.172] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/vega
log [17:01:41.699] [debug][legacy][service] Kibana server is not ready yet GET:/.
log [17:01:42.093] [info][optimize] Optimizing and caching bundles for ml, stateSessionStorageRedirect, status_page, timelion, graph, monitoring, space_selector, login, logout, dashboardViewer, apm, canvas, infra, searchguard-login, searchguard-customerror, searchguard-multitenancy, searchguard-accountinfo, searchguard-configuration and kibana. This may take a few minutes
log [17:01:45.869] [debug][legacy][proxy] "getConnections" has been called.
ops [17:01:45.924] memory: 174.9MB uptime: 0:00:23 load: [4.37 4.02 3.30] delay: 54.847
Browserslist: caniuse-lite is outdated. Please run next command `npm update caniuse-lite browserslist`
log [17:01:50.920] [debug][legacy][proxy] "getConnections" has been called.
ops [17:01:50.953] memory: 252.8MB uptime: 0:00:28 load: [4.50 4.05 3.31] delay: 33.536
log [17:01:55.939] [debug][legacy][proxy] "getConnections" has been called.
ops [17:01:56.092] memory: 260.6MB uptime: 0:00:33 load: [4.46 4.05 3.32] delay: 153.010
log [17:02:00.948] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:00.953] memory: 228.3MB uptime: 0:00:38 load: [4.26 4.01 3.31] delay: 4.598
log [17:02:06.262] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:06.278] memory: 251.8MB uptime: 0:00:44 load: [4.24 4.01 3.31] delay: 16.178
log [17:02:11.312] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:11.491] memory: 228.9MB uptime: 0:00:49 load: [4.46 4.06 3.33] delay: 178.608
log [17:02:16.396] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:16.430] memory: 336.9MB uptime: 0:00:54 load: [4.74 4.13 3.36] delay: 34.560
log [17:02:21.489] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:21.589] memory: 280.3MB uptime: 0:00:59 load: [5.00 4.19 3.38] delay: 100.217
log [17:02:27.020] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:27.197] memory: 241.0MB uptime: 0:01:04 load: [4.76 4.16 3.38] delay: 177.221
log [17:02:32.214] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:32.433] memory: 390.3MB uptime: 0:01:10 load: [4.78 4.17 3.38] delay: 218.692
log [17:02:37.217] [debug][legacy][proxy] "getConnections" has been called.
ops [17:02:37.550] memory: 317.9MB uptime: 0:01:15 load: [4.80 4.19 3.39] delay: 333.571
Also curl response for localhost:5601
curl https://localhost:5601 -vk
* About to connect() to localhost port 5601 (#0)
* Trying ::1...
* Connection refused
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5601 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=node.elastictest.com,OU=Ops,O="elastictest Com, Inc.",DC=elastictest,DC=com
* start date: May 18 19:50:26 2018 GMT
* expire date: May 15 19:50:26 2028 GMT
* common name: node.elastictest.com
* issuer: CN=signing.ca.elastictest.com,OU=CA,O="elastictest Com, Inc.",DC=elastictest,DC=com
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:5601
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< retry-after: 30
< content-type: text/html; charset=utf-8
< cache-control: no-cache
< content-length: 30
< Date: Mon, 21 Jan 2019 17:01:41 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
Kibana server is not ready yet
Thanks
Dinesh
Is there any update to this from the Elastic staff? Is this an issue with kibana optimization process?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.