Kibana weird fail after upgrade to 7.14.0

Elastic Stack Kibana
1

Today I upgraded my ES stack from 7.9.3 to 7.14.0. ES and filebeats upgraded with no issues. I have kibana installed on one of the master ES nodes. After upgrading kibana I received the output in the UI. Kibana doesn't appear to load anything.

kibana_error
kibana_error1036×916 103 KB

I dont have anything particularly weird in the kibana.yml, and those configs worked under v7.9.3. kibana.yml:

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Kibana Configs
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://REMOVED:9200", "http://REMOVED:9200", "http:/REMOVED:9200", "http://REMOVED:9200", "http://REMOVED:9201", "http://REMOVED:9200", "http://REMOVED:9201", "http://REMOVED:9200", "http://REMOVED:9201", "http://REMOVED:9200", "http://REMOVED:9201"]
elasticsearch.requestTimeout: 120000
logging.dest: /data/kibana/logs/kibana.log

There isnt much in the kibana.log. below are some of the warns. I dont think this issue is related to xpack because I stood up another server, and installed a local ES and Kibana at 7.14.0. That log produces the same warns, but Kibana loads with no issues.

{"type":"log","@timestamp":"2021-08-06T18:15:58+00:00","tags":["warning","config","deprecation"],"pid":2621,"message":"\"logging.dest\" has been deprecated and will be removed in 8.0. To set the destination moving forward, you can use the \"console\" appender in your logging configuration or define a custom one. For more details, see https://github.com/elastic/kibana/blob/master/src/core/server/logging/README.mdx"}

{"type":"log","@timestamp":"2021-08-06T19:05:30+00:00","tags":["info","plugins","taskManager"],"pid":4441,"message":"TaskManager is identified by the Kibana UUID: dd003090-bb78-4541-b679-93c291a526b3"}
{"type":"log","@timestamp":"2021-08-06T19:05:35+00:00","tags":["warning","plugins","security","config"],"pid":4441,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-06T19:05:35+00:00","tags":["warning","plugins","security","config"],"pid":4441,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2021-08-06T19:05:35+00:00","tags":["warning","plugins","reporting","config"],"pid":4441,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-06T19:05:35+00:00","tags":["warning","plugins","reporting","config"],"pid":4441,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux Red Hat Linux 7.9 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-08-06T19:05:35+00:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":4441,"message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-06T19:05:36+00:00","tags":["warning","plugins","actions","actions"],"pid":4441,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-06T19:05:36+00:00","tags":["warning","plugins","alerting","plugins","alerting"],"pid":4441,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}

can anyone point me in the right direction or have any experience with this type of error? I even tried uninstalling Kibana and reinstalling as we as installing kibana on another master node after deleting all kibana indices and still got the error.

2

Hi and welcome to our community
Could you provide some details about the installation and it's environment? do you use custom plugins in Kibana?
Thx!

3

Hello and thank you for responding! I mostly use the kibana instance to manage the indices and its cli tool. I don't use it for any dashboarding or visualizations. I do not recall ever installing any custom plugins. Is there a method to list the plugins or a directory i can check? I did create a non-default space once. but all that should be gone since while troubleshooting i deleted all indices starting with .kibana* and tried a fresh install of kibana.

the environment is probably pretty restrictive. The Elasticsearch cluster is the database for an instance of arkme (formerly moloch).

I began wondering if this was a selinux or some sort of directory restriction and am currently looking at file permissions. The Elasticsearch nodes do not appear to have any issues.

4

ah. i found the kibana-plugin script. I do not have any plugins installed?

/usr/share/kibana/bin/kibana-plugin list
No plugins installed.

i also found the plugins/data/public directory

# ls -alh /usr/share/kibana/src/plugins/data/public/
total 128K
drwxr-xr-x. 13 root root  260 Aug  9 12:24 .
drwxr-xr-x.  6 root root  138 Aug  9 12:24 ..
drwxr-xr-x.  3 root root   21 Aug  6 16:17 actions
drwxr-xr-x.  3 root root   23 Aug  6 16:17 autocomplete
drwxr-xr-x.  2 root root    6 Jul 31  2020 autocomplete_provider
drwxr-xr-x.  4 root root   37 Aug  6 16:17 field_formats
drwxr-xr-x.  2 root root    6 Jul 31  2020 field_formats_provider
drwxr-xr-x.  6 root root   72 Aug  6 16:17 index_patterns
-rw-r--r--.  1 root root  112 Jul 29 20:09 index.scss
-rw-r--r--.  1 root root 124K Jul 29 20:09 public.api.md
drwxr-xr-x.  8 root root  115 Aug  6 16:17 query
drwxr-xr-x. 10 root root  134 Aug  6 16:17 search
drwxr-xr-x.  2 root root    6 Jul 31  2020 suggestions_provider
drwxr-xr-x. 11 root root  234 Aug  9 12:24 ui
drwxr-xr-x.  3 root root   34 Aug  6 16:15 utils

is it missing the js files? ie index.js, plugin.js, services.js, stubs.js, and types.js?

5

@kleckns not sure if that is the problem or how it had happened but you could be missing or have a corrupt content for the data plugin. I would suggest if you can download the tar linux version of kibana 7.14 from https://artifacts.elastic.co/downloads/kibana/kibana-7.14.0-linux-x86_64.tar.gz, untar it and just replace the src/plugins/data folder (located at your Kibana installation folder) with the one present in the tar archive I sent you.

Cheers

6

thank you. i will give that a try.

7

unfortunately that did not resolve the issue.

I continued troubleshooting and deleted the kibana indices and then started kibana again. below are the longs since that restart

{"type":"log","@timestamp":"2021-08-09T14:49:56+00:00","tags":["info","plugins-service"],"pid":26360,"message":"Plugin \"metricsEntities\" is disabled."}
{"type":"log","@timestamp":"2021-08-09T14:49:56+00:00","tags":["warning","config","deprecation"],"pid":26360,"message":"\"logging.dest\" has been deprecated and will be removed in 8.0. To set the destination moving forward, yo
u can use the \"console\" appender in your logging configuration or define a custom one. For more details, see https://github.com/elastic/kibana/blob/master/src/core/server/logging/README.mdx"}
{"type":"log","@timestamp":"2021-08-09T14:49:56+00:00","tags":["warning","config","deprecation"],"pid":26360,"message":"plugins.scanDirs is deprecated and is no longer used"}
{"type":"log","@timestamp":"2021-08-09T14:49:56+00:00","tags":["warning","config","deprecation"],"pid":26360,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifi
cations to work in 8.0.\""}
{"type":"log","@timestamp":"2021-08-09T14:49:56+00:00","tags":["warning","config","deprecation"],"pid":26360,"message":"\"xpack.reporting.roles\" is deprecated. Granting reporting privilege through a \"reporting_user\" role wi
ll not be supported starting in 8.0. Please set \"xpack.reporting.roles.enabled\" to \"false\" and grant reporting privileges to users using Kibana application privileges **Management > Security > Roles**."}
{"type":"log","@timestamp":"2021-08-09T14:49:56+00:00","tags":["info","http","server","NotReady"],"pid":26360,"message":"http server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2021-08-09T14:49:57+00:00","tags":["info","plugins-system"],"pid":26360,"message":"Setting up [106] plugins: [translations,taskManager,licensing,globalSearch,globalSearchProviders,banners,licenseApi
Guard,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,screenshotMode,telemetry,newsfeed,mapsEms,mapsLegacy,legacyExport,kibanaLegacy,embeddable,uiAct
ionsEnhanced,expressions,charts,esUiShared,bfetch,data,savedObjects,visualizations,visTypeXy,visTypeVislib,visTypeTimelion,features,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,presen
tationUtil,timelion,home,searchprofiler,painlessLab,grokdebugger,graph,visTypeVega,management,watcher,licenseManagement,indexPatternManagement,advancedSettings,discover,discoverEnhanced,dashboard,dashboardEnhanced,visualize,vi
sTypeTimeseries,savedObjectsManagement,spaces,security,transform,savedObjectsTagging,lens,reporting,canvas,lists,ingestPipelines,fileUpload,maps,dataVisualizer,encryptedSavedObjects,dataEnhanced,timelines,dashboardMode,cloud,u
pgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,osquery,ml,case
s,securitySolution,observability,uptime,infra,monitoring,logstash,console,apmOss,apm]"}
{"type":"log","@timestamp":"2021-08-09T14:49:57+00:00","tags":["info","plugins","taskManager"],"pid":26360,"message":"TaskManager is identified by the Kibana UUID: 5bc6d179-40ca-4e79-8ad4-15e3ae764302"}
{"type":"log","@timestamp":"2021-08-09T14:50:02+00:00","tags":["warning","plugins","security","config"],"pid":26360,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-09T14:50:02+00:00","tags":["warning","plugins","security","config"],"pid":26360,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2021-08-09T14:50:02+00:00","tags":["warning","plugins","reporting","config"],"pid":26360,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-09T14:50:03+00:00","tags":["warning","plugins","reporting","config"],"pid":26360,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux Red Hat Linux 7.9 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-08-09T14:50:03+00:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":26360,"message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-09T14:50:03+00:00","tags":["warning","plugins","actions","actions"],"pid":26360,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-09T14:50:03+00:00","tags":["warning","plugins","alerting","plugins","alerting"],"pid":26360,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-09T14:50:09+00:00","tags":["info","plugins","ruleRegistry"],"pid":26360,"message":"Write is disabled, not installing assets"}
{"type":"log","@timestamp":"2021-08-09T14:50:09+00:00","tags":["info","savedobjects-service"],"pid":26360,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-08-09T14:50:11+00:00","tags":["info","savedobjects-service"],"pid":26360,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2021-08-09T14:50:11+00:00","tags":["info","savedobjects-service"],"pid":26360,"message":"[.kibana] INIT -> CREATE_NEW_TARGET. took: 199ms."}
{"type":"log","@timestamp":"2021-08-09T14:50:11+00:00","tags":["info","savedobjects-service"],"pid":26360,"message":"[.kibana_task_manager] INIT -> CREATE_NEW_TARGET. took: 204ms."}
{"type":"log","@timestamp":"2021-08-09T14:50:12+00:00","tags":["info","savedobjects-service"],"pid":26360,"message":"[.kibana] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 380ms."}
{"type":"log","@timestamp":"2021-08-09T14:50:12+00:00","tags":["info","savedobjects-service"],"pid":26360,"message":"[.kibana_task_manager] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 428ms."}
{"type":"log","@timestamp":"2021-08-09T14:50:12+00:00","tags":["info","savedobjects-service"],"pid":26360,"message":"[.kibana] MARK_VERSION_INDEX_READY -> DONE. took: 96ms."}
{"type":"log","@timestamp":"2021-08-09T14:50:12+00:00","tags":["info","savedobjects-service"],"pid":26360,"message":"[.kibana] Migration completed after 676ms"}
{"type":"log","@timestamp":"2021-08-09T14:50:12+00:00","tags":["info","savedobjects-service"],"pid":26360,"message":"[.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 76ms."}
{"type":"log","@timestamp":"2021-08-09T14:50:12+00:00","tags":["info","savedobjects-service"],"pid":26360,"message":"[.kibana_task_manager] Migration completed after 708ms"}
{"type":"log","@timestamp":"2021-08-09T14:50:12+00:00","tags":["info","status"],"pid":26360,"message":"Kibana is now unavailable"}
{"type":"log","@timestamp":"2021-08-09T14:50:12+00:00","tags":["info","plugins-system"],"pid":26360,"message":"Starting [106] plugins: [translations,taskManager,licensing,globalSearch,globalSearchProviders,banners,licenseApiGuard,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,screenshotMode,telemetry,newsfeed,mapsEms,mapsLegacy,legacyExport,kibanaLegacy,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,savedObjects,visualizations,visTypeXy,visTypeVislib,visTypeTimelion,features,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,presentationUtil,timelion,home,searchprofiler,painlessLab,grokdebugger,graph,visTypeVega,management,watcher,licenseManagement,indexPatternManagement,advancedSettings,discover,discoverEnhanced,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,savedObjectsManagement,spaces,security,transform,savedObjectsTagging,lens,reporting,canvas,lists,ingestPipelines,fileUpload,maps,dataVisualizer,encryptedSavedObjects,dataEnhanced,timelines,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,osquery,ml,cases,securitySolution,observability,uptime,infra,monitoring,logstash,console,apmOss,apm]"}
{"type":"log","@timestamp":"2021-08-09T14:50:12+00:00","tags":["info","plugins","monitoring","monitoring"],"pid":26360,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-08-09T14:50:14+00:00","tags":["info","http","server","Kibana"],"pid":26360,"message":"http server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2021-08-09T14:50:15+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":26360,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2021-08-09T14:50:16+00:00","tags":["info","plugins","securitySolution"],"pid":26360,"message":"Dependent plugin setup complete - Starting ManifestTask"}
{"type":"log","@timestamp":"2021-08-09T14:50:16+00:00","tags":["info","plugins","reporting"],"pid":26360,"message":"Browser executable: /usr/share/kibana/x-pack/plugins/reporting/chromium/headless_shell-linux_x64/headless_shell"}
{"type":"log","@timestamp":"2021-08-09T14:50:16+00:00","tags":["warning","plugins","reporting"],"pid":26360,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"log","@timestamp":"2021-08-09T14:50:18+00:00","tags":["info","status"],"pid":26360,"message":"Kibana is now available (was unavailable)"}
8

and the log entries of me trying to load kibana:

{"type":"response","@timestamp":"2021-08-09T14:50:31+00:00","tags":[],"pid":26360,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"REDACTED:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","upgrade-insecure-requests":"1"},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"},"res":{"statusCode":302,"responseTime":79},"message":"GET / 302 79ms"}
{"type":"response","@timestamp":"2021-08-09T14:50:31+00:00","tags":[],"pid":26360,"method":"get","statusCode":302,"req":{"url":"/spaces/enter","method":"get","headers":{"host":"REDACTED:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","upgrade-insecure-requests":"1"},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"},"res":{"statusCode":302,"responseTime":27},"message":"GET /spaces/enter 302 27ms"}
{"type":"response","@timestamp":"2021-08-09T14:50:32+00:00","tags":[],"pid":26360,"method":"get","statusCode":200,"req":{"url":"/app/home","method":"get","headers":{"host":"REDACTED:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","upgrade-insecure-requests":"1"},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"},"res":{"statusCode":200,"responseTime":149,"contentLength":135922},"message":"GET /app/home 200 149ms - 132.7KB"}
{"type":"response","@timestamp":"2021-08-09T14:50:32+00:00","tags":["api"],"pid":26360,"method":"get","statusCode":304,"req":{"url":"/bootstrap.js","method":"get","headers":{"host":"REDACTED:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://REDACTED:5601/app/home","connection":"keep-alive","if-none-match":"\"0849da22c774da248402ab433de9a6f9e4569d8c-gzip\""},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","referer":"http://REDACTED:5601/app/home"},"res":{"statusCode":304,"responseTime":28,"contentLength":17301},"message":"GET /bootstrap.js 304 28ms - 16.9KB"}
{"type":"response","@timestamp":"2021-08-09T14:50:35+00:00","tags":[],"pid":26360,"method":"get","statusCode":304,"req":{"url":"/translations/en.json","method":"get","headers":{"host":"REDACTED:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://REDACTED:5601/app/home","origin":"http://REDACTED:5601","connection":"keep-alive","if-none-match":"\"37992637719f97813c3068cfbf877b2d3bb43b97\""},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","referer":"http://REDACTED:5601/app/home"},"res":{"statusCode":304,"responseTime":11,"contentLength":29},"message":"GET /translations/en.json 304 11ms - 29.0B"}
9

when I upgraded to 7.14 I had quite a few problem with kibana as well.
mainly it was to do with yml file

what is your kibana.yml file looks like. post it here without all comment section

10

I dont have much in my kibana.yml file. The ES hostnames are redacted, but I have 3 master nodes and 4 data nodes. I've tried the server host on 0.0.0.0 as well as the ip address. I was logging to a custom directory, but the new systemd service file defines the logging dir and ignores that setting.

server.host: "0.0.0.0"
elasticsearch.hosts: ["http://MASTER-NODE1:9200", "http://MASTER-NODE2:9200", "http://MASTER-NODE3:9200", "http://DATA-NODE1:9200", "http://DATA-NODE2:9200", "http://DATA-NODE3:9200", "http://DATA-NODE4:9200"]
#elasticsearch.requestTimeout: 120000
#logging.dest: /data/kibana/logs/kibana.log
11

@kleckns from the last logs you posted, looks like we are one step forward considering the error you were having. If you still have Kibana loading and you can still see a log stating that Kibana is now available like the one you had:

{"type":"log","@timestamp":"2021-08-09T14:50:18+00:00","tags":["info","status"],"pid":26360,"message":"Kibana is now available (was unavailable)"}

What is it happening when you try to load Kibana on the browser?

Cheers

12

i think that log entry corresponds with the kibana UI stating "Kibana server is not ready yet". i see that in the logs after i restart the service.

13

I believe your master-node1 is the kibana node and if yes then

I will do this as test
server.host: "master-node1"
elasticsearch.hosts:["http://MASTER-NODE1:9200"]

and you saying you don't have anything else in your file?

I believe you need
elasticserarch.username: "elastic"
elasticsearch.password: "password for above"

14

@kleckns could you provide the entire and clean startup logging of your Kibana instance? I wonder if it is now blocked on migrations.

Cheers

15

@tiagocosta Do you want me to restart kibana and provide the /var/log/kibana.log from startup? I can grab that in a min

16

@elasticforme i was not running xpack on that ES cluster, so never configured the username and pass in kibana v7.9.3. Is that enabled by default on the newer version and now I need to generate those?

oh. and yes. kibana is running on master-node1. I have the 3 master nodes and the 4 data nodes defined in the elasticsearch.hosts array.

17

@tiagocosta Do you want me to restart kibana and provide the /var/log/kibana.log from startup? I can grab that in a min

@kleckns Yes please if you can!

Cheers

18

@tiagocosta
i'm over the character limit and cannot upload txt files, so i'll break up the log into a couple posts. kibana.log part 1

{"type":"log","@timestamp":"2021-08-09T16:37:16+00:00","tags":["info","plugins-service"],"pid":2390,"message":"Plugin \"metricsEntities\" is disabled."}
{"type":"log","@timestamp":"2021-08-09T16:37:16+00:00","tags":["warning","config","deprecation"],"pid":2390,"message":"\"logging.dest\" has been deprecated and will be removed in 8.0. To set the destination moving forward, you can use the \"console\" appender in your logging configuration or define a custom one. For more details, see https://github.com/elastic/kibana/blob/master/src/core/server/logging/README.mdx"}
{"type":"log","@timestamp":"2021-08-09T16:37:16+00:00","tags":["warning","config","deprecation"],"pid":2390,"message":"plugins.scanDirs is deprecated and is no longer used"}
{"type":"log","@timestamp":"2021-08-09T16:37:16+00:00","tags":["warning","config","deprecation"],"pid":2390,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2021-08-09T16:37:16+00:00","tags":["warning","config","deprecation"],"pid":2390,"message":"\"xpack.reporting.roles\" is deprecated. Granting reporting privilege through a \"reporting_user\" role will not be supported starting in 8.0. Please set \"xpack.reporting.roles.enabled\" to \"false\" and grant reporting privileges to users using Kibana application privileges **Management > Security > Roles**."}
{"type":"log","@timestamp":"2021-08-09T16:37:16+00:00","tags":["info","http","server","NotReady"],"pid":2390,"message":"http server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2021-08-09T16:37:17+00:00","tags":["info","plugins-system"],"pid":2390,"message":"Setting up [106] plugins: [translations,taskManager,licensing,globalSearch,globalSearchProviders,banners,licenseApiGuard,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,screenshotMode,telemetry,newsfeed,mapsEms,mapsLegacy,legacyExport,kibanaLegacy,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,savedObjects,visualizations,visTypeXy,visTypeVislib,visTypeTimelion,features,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,presentationUtil,timelion,home,searchprofiler,painlessLab,grokdebugger,graph,visTypeVega,management,watcher,licenseManagement,indexPatternManagement,advancedSettings,discover,discoverEnhanced,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,savedObjectsManagement,spaces,security,transform,savedObjectsTagging,lens,reporting,canvas,lists,ingestPipelines,fileUpload,maps,dataVisualizer,encryptedSavedObjects,dataEnhanced,timelines,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,osquery,ml,cases,securitySolution,observability,uptime,infra,monitoring,logstash,console,apmOss,apm]"}
{"type":"log","@timestamp":"2021-08-09T16:37:17+00:00","tags":["info","plugins","taskManager"],"pid":2390,"message":"TaskManager is identified by the Kibana UUID: 30b5e16b-928b-4e6e-be0f-90915d922582"}
{"type":"log","@timestamp":"2021-08-09T16:37:21+00:00","tags":["warning","plugins","security","config"],"pid":2390,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-09T16:37:21+00:00","tags":["warning","plugins","security","config"],"pid":2390,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2021-08-09T16:37:21+00:00","tags":["warning","plugins","reporting","config"],"pid":2390,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-09T16:37:21+00:00","tags":["warning","plugins","reporting","config"],"pid":2390,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux Red Hat Linux 7.9 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-08-09T16:37:21+00:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":2390,"message":"Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-09T16:37:22+00:00","tags":["warning","plugins","actions","actions"],"pid":2390,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-09T16:37:22+00:00","tags":["warning","plugins","alerting","plugins","alerting"],"pid":2390,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-08-09T16:37:27+00:00","tags":["info","plugins","ruleRegistry"],"pid":2390,"message":"Write is disabled, not installing assets"}
{"type":"log","@timestamp":"2021-08-09T16:37:28+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-08-09T16:37:29+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2021-08-09T16:37:29+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"[.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 124ms."}
{"type":"log","@timestamp":"2021-08-09T16:37:29+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"[.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 123ms."}
{"type":"log","@timestamp":"2021-08-09T16:37:29+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 10ms."}
{"type":"log","@timestamp":"2021-08-09T16:37:29+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 12ms."}
{"type":"log","@timestamp":"2021-08-09T16:37:29+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 16ms."}
{"type":"log","@timestamp":"2021-08-09T16:37:29+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 26ms."}
{"type":"log","@timestamp":"2021-08-09T16:37:29+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 6ms."}
{"type":"log","@timestamp":"2021-08-09T16:37:29+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 38ms."}
{"type":"log","@timestamp":"2021-08-09T16:37:30+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"[.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 102ms."}
{"type":"log","@timestamp":"2021-08-09T16:37:30+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 92ms."}
{"type":"log","@timestamp":"2021-08-09T16:37:30+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 142ms."}
{"type":"log","@timestamp":"2021-08-09T16:37:30+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"[.kibana_task_manager] Migration completed after 424ms"}
{"type":"log","@timestamp":"2021-08-09T16:37:30+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"[.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 308ms."}
{"type":"log","@timestamp":"2021-08-09T16:37:30+00:00","tags":["info","savedobjects-service"],"pid":2390,"message":"[.kibana] Migration completed after 577ms"}
{"type":"log","@timestamp":"2021-08-09T16:37:30+00:00","tags":["info","status"],"pid":2390,"message":"Kibana is now unavailable"}
{"type":"log","@timestamp":"2021-08-09T16:37:30+00:00","tags":["info","plugins-system"],"pid":2390,"message":"Starting [106] plugins: [translations,taskManager,licensing,globalSearch,globalSearchProviders,banners,licenseApiGuard,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,screenshotMode,telemetry,newsfeed,mapsEms,mapsLegacy,legacyExport,kibanaLegacy,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,savedObjects,visualizations,visTypeXy,visTypeVislib,visTypeTimelion,features,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,presentationUtil,timelion,home,searchprofiler,painlessLab,grokdebugger,graph,visTypeVega,management,watcher,licenseManagement,indexPatternManagement,advancedSettings,discover,discoverEnhanced,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,savedObjectsManagement,spaces,security,transform,savedObjectsTagging,lens,reporting,canvas,lists,ingestPipelines,fileUpload,maps,dataVisualizer,encryptedSavedObjects,dataEnhanced,timelines,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,osquery,ml,cases,securitySolution,observability,uptime,infra,monitoring,logstash,console,apmOss,apm]"}
{"type":"log","@timestamp":"2021-08-09T16:37:30+00:00","tags":["info","plugins","monitoring","monitoring"],"pid":2390,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-08-09T16:37:31+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":2390,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2021-08-09T16:37:33+00:00","tags":["info","http","server","Kibana"],"pid":2390,"message":"http server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2021-08-09T16:37:33+00:00","tags":["info","plugins","securitySolution"],"pid":2390,"message":"Dependent plugin setup complete - Starting ManifestTask"}
{"type":"log","@timestamp":"2021-08-09T16:37:34+00:00","tags":["info","plugins","reporting"],"pid":2390,"message":"Browser executable: /usr/share/kibana/x-pack/plugins/reporting/chromium/headless_shell-linux_x64/headless_shell"}
{"type":"log","@timestamp":"2021-08-09T16:37:34+00:00","tags":["warning","plugins","reporting"],"pid":2390,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"log","@timestamp":"2021-08-09T16:37:35+00:00","tags":["info","status"],"pid":2390,"message":"Kibana is now available (was unavailable)"}
19

@tiagocosta
part 2 of kibana.log

{"type":"response","@timestamp":"2021-08-09T16:37:46+00:00","tags":[],"pid":2390,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"MASTER-NODE1:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","upgrade-insecure-requests":"1","cache-control":"max-age=0"},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"},"res":{"statusCode":302,"responseTime":53},"message":"GET / 302 53ms"}
{"type":"response","@timestamp":"2021-08-09T16:37:46+00:00","tags":[],"pid":2390,"method":"get","statusCode":302,"req":{"url":"/spaces/enter","method":"get","headers":{"host":"MASTER-NODE1:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","upgrade-insecure-requests":"1","cache-control":"max-age=0"},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"},"res":{"statusCode":302,"responseTime":34},"message":"GET /spaces/enter 302 34ms"}
{"type":"response","@timestamp":"2021-08-09T16:37:46+00:00","tags":[],"pid":2390,"method":"get","statusCode":200,"req":{"url":"/app/home","method":"get","headers":{"host":"MASTER-NODE1:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","upgrade-insecure-requests":"1","cache-control":"max-age=0"},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"},"res":{"statusCode":200,"responseTime":195,"contentLength":135922},"message":"GET /app/home 200 195ms - 132.7KB"}
{"type":"response","@timestamp":"2021-08-09T16:37:47+00:00","tags":[],"pid":2390,"method":"get","statusCode":200,"req":{"url":"/42747/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.css","method":"get","headers":{"host":"MASTER-NODE1:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"text/css,*/*;q=0.1","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://MASTER-NODE1:5601/app/home","connection":"keep-alive","cache-control":"max-age=0"},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","referer":"http://MASTER-NODE1:5601/app/home"},"res":{"statusCode":200,"responseTime":50,"contentLength":13541},"message":"GET /42747/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.css 200 50ms - 13.2KB"}
{"type":"response","@timestamp":"2021-08-09T16:37:47+00:00","tags":[],"pid":2390,"method":"get","statusCode":200,"req":{"url":"/42747/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.v8.light.css","method":"get","headers":{"host":"MASTER-NODE1:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"text/css,*/*;q=0.1","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://MASTER-NODE1:5601/app/home","connection":"keep-alive","cache-control":"max-age=0"},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","referer":"http://MASTER-NODE1:5601/app/home"},"res":{"statusCode":200,"responseTime":39,"contentLength":66801},"message":"GET /42747/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.v8.light.css 200 39ms - 65.2KB"}
{"type":"response","@timestamp":"2021-08-09T16:37:47+00:00","tags":[],"pid":2390,"method":"get","statusCode":304,"req":{"url":"/node_modules/@kbn/ui-framework/dist/kui_light.css","method":"get","headers":{"host":"MASTER-NODE1:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"text/css,*/*;q=0.1","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://MASTER-NODE1:5601/app/home","connection":"keep-alive","if-modified-since":"Thu, 29 Jul 2021 20:09:30 GMT","if-none-match":"\"47af9b8f6e4640c400d5d0b64bd8c8095c9bbcf5-gzip\"","cache-control":"max-age=0"},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","referer":"http://MASTER-NODE1:5601/app/home"},"res":{"statusCode":304,"responseTime":41,"contentLength":205},"message":"GET /node_modules/@kbn/ui-framework/dist/kui_light.css 304 41ms - 205.0B"}
{"type":"response","@timestamp":"2021-08-09T16:37:47+00:00","tags":[],"pid":2390,"method":"get","statusCode":304,"req":{"url":"/ui/legacy_light_theme.css","method":"get","headers":{"host":"MASTER-NODE1:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"text/css,*/*;q=0.1","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://MASTER-NODE1:5601/app/home","connection":"keep-alive","if-modified-since":"Thu, 29 Jul 2021 20:09:32 GMT","if-none-match":"\"4345f9a3b9a243edd0b81b913ba1901bce549a2b-gzip\"","cache-control":"max-age=0"},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","referer":"http://MASTER-NODE1:5601/app/home"},"res":{"statusCode":304,"responseTime":38,"contentLength":206},"message":"GET /ui/legacy_light_theme.css 304 38ms - 206.0B"}
{"type":"response","@timestamp":"2021-08-09T16:37:47+00:00","tags":["api"],"pid":2390,"method":"get","statusCode":304,"req":{"url":"/bootstrap.js","method":"get","headers":{"host":"MASTER-NODE1:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://MASTER-NODE1:5601/app/home","connection":"keep-alive","if-none-match":"\"0849da22c774da248402ab433de9a6f9e4569d8c-gzip\"","cache-control":"max-age=0"},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","referer":"http://MASTER-NODE1:5601/app/home"},"res":{"statusCode":304,"responseTime":46,"contentLength":17301},"message":"GET /bootstrap.js 304 46ms - 16.9KB"}
{"type":"response","@timestamp":"2021-08-09T16:37:52+00:00","tags":[],"pid":2390,"method":"get","statusCode":304,"req":{"url":"/translations/en.json","method":"get","headers":{"host":"MASTER-NODE1:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://MASTER-NODE1:5601/app/home","origin":"http://MASTER-NODE1:5601","connection":"keep-alive","if-none-match":"\"37992637719f97813c3068cfbf877b2d3bb43b97\""},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","referer":"http://MASTER-NODE1:5601/app/home"},"res":{"statusCode":304,"responseTime":12,"contentLength":29},"message":"GET /translations/en.json 304 12ms - 29.0B"}
{"type":"response","@timestamp":"2021-08-09T16:37:59+00:00","tags":[],"pid":2390,"method":"get","statusCode":200,"req":{"url":"/status","method":"get","headers":{"host":"MASTER-NODE1:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","upgrade-insecure-requests":"1"},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"},"res":{"statusCode":200,"responseTime":117,"contentLength":135922},"message":"GET /status 200 117ms - 132.7KB"}
{"type":"response","@timestamp":"2021-08-09T16:38:00+00:00","tags":["api"],"pid":2390,"method":"get","statusCode":304,"req":{"url":"/bootstrap.js","method":"get","headers":{"host":"MASTER-NODE1:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://MASTER-NODE1:5601/status","connection":"keep-alive","if-none-match":"\"0849da22c774da248402ab433de9a6f9e4569d8c-gzip\""},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","referer":"http://MASTER-NODE1:5601/status"},"res":{"statusCode":304,"responseTime":36,"contentLength":17301},"message":"GET /bootstrap.js 304 36ms - 16.9KB"}
{"type":"response","@timestamp":"2021-08-09T16:38:02+00:00","tags":[],"pid":2390,"method":"get","statusCode":304,"req":{"url":"/translations/en.json","method":"get","headers":{"host":"MASTER-NODE1:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://MASTER-NODE1:5601/status","origin":"http://MASTER-NODE1:5601","connection":"keep-alive","if-none-match":"\"37992637719f97813c3068cfbf877b2d3bb43b97\""},"remoteAddress":"10.5.210.11","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0","referer":"http://MASTER-NODE1:5601/status"},"res":{"statusCode":304,"responseTime":4,"contentLength":29},"message":"GET /translations/en.json 304 4ms - 29.0B"}
20

try with simplest one, only one master node.

it is also complaining about this. why not test it out with this.

xpack.security.encryptionKey: "put something 32 character or more string here"