Kibana crashes

jordan_sumitomo · May 2, 2021, 11:40am

I've recently started trying out elasticsearch and kibana
but kibana crashes

Here's my kibana log

  log   [17:39:57.905] [info][plugins-service] Plugin "osquery" is disabled.
  log   [17:39:58.029] [warning][config][deprecation] Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0."
  log   [17:39:58.503] [info][plugins-system] Setting up [100] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsLegacy,kibanaLegacy,translations,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeVislib,visTypeMetric,visTypeVega,visTypeTimelion,features,licenseManagement,watcher,canvas,visTypeTagcloud,visTypeTable,visTypeMarkdown,tileMap,regionMap,visTypeXy,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,discover,discoverEnhanced,savedObjectsManagement,spaces,security,reporting,savedObjectsTagging,maps,lens,lists,dataEnhanced,encryptedSavedObjects,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,ml,securitySolution,case,infra,monitoring,logstash,apm,uptime]
  log   [17:39:58.507] [info][plugins][taskManager] TaskManager is identified by the Kibana UUID: 8f34d202-4e61-4e29-923c-304caca6dca8
  log   [17:39:59.089] [warning][config][plugins][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
  log   [17:39:59.090] [warning][config][plugins][security] Session cookies will be transmitted over insecure connections. This is not recommended.
  log   [17:39:59.142] [warning][config][plugins][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
  log   [17:39:59.185] [info][config][plugins][reporting] Chromium sandbox provides an additional layer of protection, and is supported for Win32 OS. Automatically enabling Chromium sandbox.
  log   [17:39:59.195] [warning][encryptedSavedObjects][plugins] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
  log   [17:39:59.214] [warning][fleet][plugins] Fleet APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
  log   [17:39:59.316] [warning][actions][actions][plugins] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
  log   [17:39:59.331] [warning][alerting][alerts][plugins][plugins] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
  log   [17:39:59.513] [info][monitoring][monitoring][plugins] config sourced from: production cluster
  log   [17:39:59.882] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
  log   [17:39:59.910] [error][elasticsearch] Request error, retrying
GET http://localhost:9200/_xpack?accept_enterprise=true => connect ECONNREFUSED 127.0.0.1:9200
  log   [17:39:59.914] [warning][elasticsearch] Unable to revive connection: http://localhost:9200/
  log   [17:39:59.915] [warning][elasticsearch] No living connections
  log   [17:39:59.916] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to Error: No Living connections error
  log   [17:39:59.919] [warning][monitoring][monitoring][plugins] X-Pack Monitoring Cluster Alerts will not be available: No Living connections
  log   [17:39:59.957] [error][savedobjects-service] Unable to retrieve version information from Elasticsearch nodes.
  log   [17:40:05.069] [info][savedobjects-service] Starting saved objects migrations
  log   [17:40:05.124] [info][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH. took: 19ms.
  log   [17:40:05.142] [info][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH. took: 39ms.
  log   [17:40:05.292] [info][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH -> UPDATE_TARGET_MAPPINGS. took: 168ms.
  log   [17:40:05.309] [info][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH -> UPDATE_TARGET_MAPPINGS. took: 167ms.
  log   [17:40:05.315] [error][savedobjects-service] [.kibana_task_manager] [cluster_block_exception]: index [.kibana_task_manager_7.12.1_001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
  log   [17:40:05.317] [error][savedobjects-service] [.kibana_task_manager] migration failed, dumping execution log:
  log   [17:40:05.323] [info][savedobjects-service] [.kibana_task_manager] INIT RESPONSE
  log   [17:40:05.325] [info][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH
  log   [17:40:05.327] [info][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH RESPONSE
  log   [17:40:05.328] [info][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH -> UPDATE_TARGET_MAPPINGS
  log   [17:40:05.330] [fatal][root] Error: Unable to complete saved object migrations for the [.kibana_task_manager] index. Please check the health of your Elasticsearch cluster and try again. Error: [cluster_block_exception]: index [.kibana_task_manager_7.12.1_001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
    at migrationStateActionMachine (C:\Users\Vedant\projects\elastic\kibana-7.12.1-windows-x86_64\kibana-7.12.1-windows-x86_64\src\core\server\saved_objects\migrationsv2\migrations_state_action_machine.js:138:13)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Promise.all (index 1)
    at SavedObjectsService.start (C:\Users\Vedant\projects\elastic\kibana-7.12.1-windows-x86_64\kibana-7.12.1-windows-x86_64\src\core\server\saved_objects\saved_objects_service.js:163:7)
    at Server.start (C:\Users\Vedant\projects\elastic\kibana-7.12.1-windows-x86_64\kibana-7.12.1-windows-x86_64\src\core\server\server.js:283:31)
    at Root.start (C:\Users\Vedant\projects\elastic\kibana-7.12.1-windows-x86_64\kibana-7.12.1-windows-x86_64\src\core\server\root\index.js:58:14)
    at bootstrap (C:\Users\Vedant\projects\elastic\kibana-7.12.1-windows-x86_64\kibana-7.12.1-windows-x86_64\src\core\server\bootstrap.js:100:5)
    at Command.<anonymous> (C:\Users\Vedant\projects\elastic\kibana-7.12.1-windows-x86_64\kibana-7.12.1-windows-x86_64\src\cli\serve\serve.js:169:5)
  log   [17:40:05.340] [info][plugins-system] Stopping all plugins.
  log   [17:40:05.342] [info][kibana-monitoring][monitoring][monitoring][plugins] Monitoring stats collection is stopped  log   [17:40:05.348] [error][savedobjects-service] [.kibana] [cluster_block_exception]: index [.kibana_7.12.1_001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
  log   [17:40:05.352] [error][savedobjects-service] [.kibana] migration failed, dumping execution log:
  log   [17:40:05.352] [info][savedobjects-service] [.kibana] INIT RESPONSE
  log   [17:40:05.357] [info][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH
  log   [17:40:05.362] [info][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH RESPONSE
  log   [17:40:05.365] [info][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH -> UPDATE_TARGET_MAPPINGS
  log   [17:40:29.907] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to Error: Cluster client cannot be used after it has been closed. error
  log   [17:40:35.349] [warning][plugins-system] "eventLog" plugin didn't stop in 30sec., move on to the next.

I am using elasticsearch-7.12.1-windows-x86_64,kibana-7.12.1-windows-x86_64
and run both using a bat script

    echo  Strating ELK Stack
START cmd /C C:\Users\Admin\projects\elastic\elasticsearch-7.12.1-windows-x86_64\elasticsearch-7.12.1\bin\elasticsearch.bat
echo Starting kibana
START cmd /C C:\Users\Admin\projects\elastic\kibana-7.12.1-windows-x86_64\kibana-7.12.1-windows-x86_64\bin\kibana.bat

Can anyone provide a solution to My Problem

rashmi · May 3, 2021, 5:17am

I see TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
Elasticsearch considers the available disk space before deciding whether to allocate new shards, relocate shards away or block all index write operations on a data node based on a different threshold of this error. This is because Elasticsearch indices consist of different shards which are persisted on data nodes and low disk space can cause issues.Try these steps:

Delete unused indices.
Attach external disk or increase the disk used by the data node.
Manually move shards away from the node using cluster reroute API.
Reduce replicas count to 1 (if replicas > 1).
Add new data nodes.

Thanks
Rashmi

jordan_sumitomo · May 3, 2021, 6:24am

Thanks for the help it worked.I moved it to another drive I've just gotten into elk stack I didn't have any data imported but it was the low disk space .can you provide any tips for debugging in the elk stack also resources on how to use nosql databases in the elk stack and how to containerize elk and get it running on a k8s cluster
Thank's for the reply

rashmi · May 3, 2021, 2:41pm

Glad it worked. With Elastic Cloud on Kubernetes (ECK) you can extend the basic Kubernetes orchestration capabilities to easily deploy, secure, upgrade your Elasticsearch cluster, and much more. Have you taken a look at Quickstart | Elastic Cloud on Kubernetes [master] | Elastic

Elasticsearch as a NoSQL Database | Elastic Blog - this provides the explanation in detail.

Let us know if you need more information.

Thanks
Rashmi

system · May 31, 2021, 2:41pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.