Kibana server is not ready yet on elastic 8.0

Elastic Stack Kibana
1

Hello community, first of all many many congrats on elastic 8.0 release.
I have setup Elasticsearch and kibana 8.0, Elasticsearch is running perfectly fine, below is the output of curl -X GET https://localhost:9200 -u elastic:my-password-here -k
{ "name" : "ip-172-31-72-165.ec2.internal", "cluster_name" : "elasticsearch", "cluster_uuid" : "ZFx6oZ4oQq-Xn8yPKDwTOA", "version" : { "number" : "8.0.0", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "1b6a7ece17463df5ff54a3e1302d825889aa1161", "build_date" : "2022-02-03T16:47:57.507843096Z", "build_snapshot" : false, "lucene_version" : "9.0.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" }

Now when I start kibana with elastic default password auto generated at the time of Elasticsearch installation, it shows me blank page in browwer, while commenting both the username and password for Elasticsearch in kibana.yml and providing the token, it start loading kibana page with logo but then shows, "Kibana server is not ready yet."

server.host : "0.0.0.0"
Elasticsearch.hosts : ["https://localhost:9200"]
Elasticsearch.serviceAccountToken : "token-here"
Elasticsearch.ssl.verificationMode : 'none'

Kibana logs
set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.encryptedSavedObjects"},"process":{"pid":5730}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-16T21:33:11.661+00:00","message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.actions"},"process":{"pid":5730}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-16T21:33:11.698+00:00","message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.alerting"},"process":{"pid":5730}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-16T21:33:11.728+00:00","message":"Installing common resources shared between all indices","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":5730}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-16T21:33:12.580+00:00","message":"Chromium sandbox provides an additional layer of protection, and is supported for Linux Amazon Linux 2 (Karoo) OS. Automatically enabling Chromium sandbox.","log":{"level":"INFO","logger":"plugins.screenshotting.config"},"process":{"pid":5730}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-16T21:33:13.561+00:00","message":"Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate with provided credentials and anonymous access is not allowed for this request","log":{"level":"ERROR","logger":"Elasticsearch-service"},"process":{"pid":5730}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-16T21:33:16.877+00:00","message":"Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell","log":{"level":"INFO","logger":"plugins.screenshotting.chromium"},"process":{"pid":5730}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-16T21:33:18.152+00:00","message":"Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 127.0.0.1:9200","log":{"level":"ERROR","logger":"Elasticsearch-service"},"process":{"pid":5730}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-16T21:33:52.900+00:00","message":"Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate with provided credentials and anonymous access is not allowed for this request","log":{"level":"ERROR","logger":"Elasticsearch-service"},"process":{"pid":5730}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-16T21:53:11.729+00:00","message":"Timeout: it took more than 1200000ms","error":{"message":"Timeout: it took more than 1200000ms","type":"Error","stack_trace":"Error: Timeout: it took more than 1200000ms\n at Timeout._onTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:54:20)\n at listOnTimeout (node:internal/timers:557:17)\n at processTimers (node:internal/timers:500:7)"},"log":{"level":"ERROR","logger":"plugins.ruleRegistry"},"process":{"pid":5730}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-16T21:53:11.732+00:00","message":"Failure installing common resources shared between all indices. Timeout: it took more than 1200000ms","error":{"message":"Failure installing common resources shared between all indices. Timeout: it took more than 1200000ms","type":"Error","stack_trace":"Error: Failure installing common resources shared between all indices. Timeout: it took more than 1200000ms\n at ResourceInstaller.installWithTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:63:13)\n at ResourceInstaller.installCommonResources (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:77:5)"},"log":{"level":"ERROR","logger":"plugins.ruleRegistry"},"process":{"pid":5730}}

Elasticsearch logs
[2022-02-16T21:33:45,029][INFO ][o.e.t.n.NettyAllocator ] [ip-172-31-72-165.ec2.internal] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2022-02-16T21:33:45,240][INFO ][o.e.d.DiscoveryModule ] [ip-172-31-72-165.ec2.internal] using discovery type [zen] and seed hosts providers [settings]
[2022-02-16T21:33:47,384][INFO ][o.e.n.Node ] [ip-172-31-72-165.ec2.internal] initialized
[2022-02-16T21:33:47,385][INFO ][o.e.n.Node ] [ip-172-31-72-165.ec2.internal] starting ...
[2022-02-16T21:33:47,407][INFO ][o.e.x.s.c.f.PersistentCache] [ip-172-31-72-165.ec2.internal] persistent cache index loaded
[2022-02-16T21:33:47,408][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [ip-172-31-72-165.ec2.internal] deprecation component started
[2022-02-16T21:33:47,584][INFO ][o.e.t.TransportService ] [ip-172-31-72-165.ec2.internal] publish_address {172.31.72.165:9300}, bound_addresses {[::]:9300}
[2022-02-16T21:33:48,067][INFO ][o.e.b.BootstrapChecks ] [ip-172-31-72-165.ec2.internal] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-02-16T21:33:48,106][INFO ][o.e.c.c.Coordinator ] [ip-172-31-72-165.ec2.internal] cluster UUID [ZFx6oZ4oQq-Xn8yPKDwTOA]
[2022-02-16T21:33:48,356][INFO ][o.e.c.s.MasterService ] [ip-172-31-72-165.ec2.internal] elected-as-master ([1] nodes joined)[{ip-172-31-72-165.ec2.internal}{7-QJqFl9SIKGHrGrAeCk1Q}{0wR7CZu5R22gVJ8GLP-MVA}{172.31.72.165}{172.31.72.165:9300}{cdfhilmrstw} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 10, version: 129, delta: master node changed {previous , current [{ip-172-31-72-165.ec2.internal}{7-QJqFl9SIKGHrGrAeCk1Q}{0wR7CZu5R22gVJ8GLP-MVA}{172.31.72.165}{172.31.72.165:9300}{cdfhilmrstw}]}
[2022-02-16T21:33:48,684][INFO ][o.e.c.s.ClusterApplierService] [ip-172-31-72-165.ec2.internal] master node changed {previous , current [{ip-172-31-72-165.ec2.internal}{7-QJqFl9SIKGHrGrAeCk1Q}{0wR7CZu5R22gVJ8GLP-MVA}{172.31.72.165}{172.31.72.165:9300}{cdfhilmrstw}]}, term: 10, version: 129, reason: Publication{term=10, version=129}
[2022-02-16T21:33:49,243][INFO ][o.e.h.AbstractHttpServerTransport] [ip-172-31-72-165.ec2.internal] publish_address {172.31.72.165:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {172.31.72.165:9200}
[2022-02-16T21:33:49,244][INFO ][o.e.n.Node ] [ip-172-31-72-165.ec2.internal] started
[2022-02-16T21:33:49,287][INFO ][o.e.l.LicenseService ] [ip-172-31-72-165.ec2.internal] license [adc6bd23-ba65-4fd2-8650-4d244b60e4ed] mode [basic] - valid
[2022-02-16T21:33:49,289][INFO ][o.e.x.s.a.Realms ] [ip-172-31-72-165.ec2.internal] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2022-02-16T21:33:49,311][INFO ][o.e.g.GatewayService ] [ip-172-31-72-165.ec2.internal] recovered [2] indices into cluster_state
[2022-02-16T21:33:50,131][INFO ][o.e.c.r.a.AllocationService] [ip-172-31-72-165.ec2.internal] current.health="GREEN" message="Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.security-7][0], [.geoip_databases][0]]])." previous.health="RED" reason="shards started [[.security-7][0], [.geoip_databases][0]]"
[2022-02-16T21:33:50,934][INFO ][o.e.i.g.DatabaseNodeService] [ip-172-31-72-165.ec2.internal] successfully loaded geoip database file [GeoLite2-Country.mmdb]
[2022-02-16T21:33:51,066][INFO ][o.e.i.g.DatabaseNodeService] [ip-172-31-72-165.ec2.internal] successfully loaded geoip database file [GeoLite2-ASN.mmdb]
[2022-02-16T21:33:52,436][INFO ][o.e.i.g.DatabaseNodeService] [ip-172-31-72-165.ec2.internal] successfully loaded geoip database file [GeoLite2-City.mmdb]

2

I just created the token using below command,
/usr/share/Elasticsearch/bin/Elasticsearch-create-enrollment-token -s kibana

I didn't copy it to any config file, what might be the next step if I am missing,

3

You can reinstall kibana. The first time it starts, you can navigate to the kibana URL, enter the enrollment token you generated in Elasticsearch and it will configure itself. Note that you would need to re-generate an enrollment token as these are valid only for 30 mins

4

Hello @ikakavas, I have reinstalled the kibana server, restarted the Elasticsearch and then regenerated the token and started kibana, on loading in browser, still it says, "Kibana server is not ready yet"

here are the logs;

{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T08:22:25.750+00:00","message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.reporting.config"},"process":{"pid":15356}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T08:22:25.752+00:00","message":"Found 'server.host: \"0.0.0.0\"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.","log":{"level":"WARN","logger":"plugins.reporting.config"},"process":{"pid":15356}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T08:22:25.765+00:00","message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.encryptedSavedObjects"},"process":{"pid":15356}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T08:22:25.784+00:00","message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.actions"},"process":{"pid":15356}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T08:22:25.810+00:00","message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.","log":{"level":"WARN","logger":"plugins.alerting"},"process":{"pid":15356}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T08:22:25.840+00:00","message":"Installing common resources shared between all indices","log":{"level":"INFO","logger":"plugins.ruleRegistry"},"process":{"pid":15356}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T08:22:26.663+00:00","message":"Chromium sandbox provides an additional layer of protection, and is supported for Linux Amazon Linux 2 (Karoo) OS. Automatically enabling Chromium sandbox.","log":{"level":"INFO","logger":"plugins.screenshotting.config"},"process":{"pid":15356}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T08:22:29.725+00:00","message":"Unable to retrieve version information from Elasticsearch nodes. self signed certificate in certificate chain","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":15356}}
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T08:22:30.364+00:00","message":"Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell","log":{"level":"INFO","logger":"plugins.screenshotting.chromium"},"process":{"pid":15356}}

one more thing, after installing kibana I have just changed the below lines, all the rest are default;

elasticsearch.hosts: ["https://localhost:9200"]` server.host: "0.0.0.0

is their any other changes required regarding ssl?
I have also tried
elasticsearch.ssl.verificationMode: 'none'

5

Don't change anything after you reinstall kibana. Kibana will configure itself with the enrollment token. You won't have to touch the configuration files ! :slight_smile:

By adding some of the config yourself, you are indicating that you want to do the configuration yourself and this is why we're not triggering the automatic configuration

6

you mean,

  1. install Elasticsearch
  2. install kibana
  3. start Elasticsearch and generate token for kibana
  4. start kibana
  5. open in browser and provide the token there?
7

Hello Ikakavas, I created a new ec2 instance on AWS, the output is the same, this time, I just installed Elasticsearch with default config, I didn't change any line just started Elasticsearch, then installed kibana with default config, just created token and then on starting kibana I can not see anything in the browser, not any single line. Although I can see output of doing curl Elasticsearch with default username and password.

8

then installed kibana with default config, just created token and then on starting kibana I can not see anything in the browser, not any single line.

  • What is the output from starting Kibana ?
  • What is in the kibana logs ?
  • Which URL did you try to connect to Kibana at ?

Yes, this is also described in detail in our documentation here: Start the Elastic Stack with security enabled | Elasticsearch Guide [8.0] | Elastic

9

  • What is the output from starting Kibana ?
    nothing show in the browser

  • Which URL did you try to connect to Kibana at ?
    my ec2 instance public ip with 5601 port

  • What is in the kibana logs ?
    Kibana logs

{"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T12:23:36.936+00:00","message":"http server running at [http://localhost:5601](http://localhost:5601/)","log":{"level":"INFO","logger":"http.server.Preboot"},"process":{"pid":4274}} {"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T12:23:36.985+00:00","message":"Setting up [1] plugins: [interactiveSetup]","log":{"level":"INFO","logger":"plugins-system.preboot"},"process":{"pid":4274}} {"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T12:23:36.988+00:00","message":"\"interactiveSetup\" plugin is holding setup: Validating Elasticsearch connection configuration…","log":{"level":"INFO","logger":"preboot"},"process":{"pid":4274}} {"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T12:23:37.035+00:00","message":"Holding setup until preboot stage is completed.","log":{"level":"INFO","logger":"root"},"process":{"pid":4274}} {"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T12:39:36.572+00:00","message":"Stopping all plugins.","log":{"level":"INFO","logger":"plugins-system.preboot"},"process":{"pid":4274}} {"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T12:39:48.814+00:00","message":"Plugin \"metricsEntities\" is disabled.","log":{"level":"INFO","logger":"plugins-service"},"process":{"pid":5378}} {"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T12:39:48.927+00:00","message":"http server running at [http://localhost:5601](http://localhost:5601/)","log":{"level":"INFO","logger":"http.server.Preboot"},"process":{"pid":5378}} {"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T12:39:48.978+00:00","message":"Setting up [1] plugins: [interactiveSetup]","log":{"level":"INFO","logger":"plugins-system.preboot"},"process":{"pid":5378}} {"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T12:39:48.981+00:00","message":"\"interactiveSetup\" plugin is holding setup: Validating Elasticsearch connection configuration…","log":{"level":"INFO","logger":"preboot"},"process":{"pid":5378}} {"ecs":{"version":"8.0.0"},"@timestamp":"2022-02-17T12:39:49.020+00:00","message":"Holding setup until preboot stage is completed.","log":{"level":"INFO","logger":"root"},"process":{"pid":5378}}
10

Thanks @ikakavas, kibana is now connected with Elasticsearch now, we were digging the place.
putting the instance private ip into /etc/hosts on the Elasticsearch machine solve my issue.

is there any blog options with elastic.co, so I can write a blog on it and post for helping purpose.

11

How did you install kibana and how do you start it ? I was more interested for output from when you start kibana, not in the browser. Can you share some more details please

putting the instance private ip into /etc/hosts on the Elasticsearch machine solve my issue.

Apologies but I’d like to understand a bit more about this issue that you were having and why and how adding the IP to /etc/hosts might have helped.
Would you be so kind as to share with us in detail

  • all the actions you made
  • how did you install elasticsearch ( the actual command you use )
  • how did you install kibana ( the actual command you use )
  • how did you start elasticsearch ( the actual command you use )
  • how did you start kibana ( the actual command you use and the output )

Thanks a lot for your feedback!

12

I can post a blog on it if there is any option at your site.

or I can also share a link to my confluence how to. Do let me know.

13

Please post the details here, for the benefit of the entire community. If this is a bug on our side or a well known issue that you’ve found a solution for, we can make sure the information gets properly circulated but for now we’re still trying to understand what your problem was and how you fixed it.

Thanks

1 Like
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.