Can't turn on monitoring

avnere · November 28, 2021, 8:41am

Hi,

I have installed Kibana for the new cluster.

When going to Stack Monitoring. It shows “We couldn't activate monitoring”

In addition log prints “Unable to bulk upload the stats payload to the local cluster”

Dev tools and Index management are working properly.

I tried the following:

Checked kibana status (http://10.0.59.155:5601/status) --> Seems OK.
Checked if .monitoring index exist --> does not exist.
Checked if monitoring.collection.enabled=true --> true
Restarted the nodes and kibana.

kibana

Is it familiar to you?

Can you assist on it?

Thanks,

Avner

warkolm · November 28, 2021, 9:29pm

Welcome to our community!

Can you please post the entire Kibana and Elasticsearch log for us?

avnere · November 29, 2021, 12:47pm

Kibana Log:

Last login: Thu Nov 25 17:49:02 2021 from ae-dev-win10.ad.shared
[xservice@iky-ed-lx-st05 ~]$ cd /storage/elastic/kibana/kibana-7.13.1-linux-x86_64/bin/
[xservice@iky-ed-lx-st05 bin]$ ./kibana
log [17:58:27.673] [info][plugins-service] Plugin "timelines" is disabled.
log [17:58:27.753] [warning][config][deprecation] plugins.scanDirs is deprecated and is no longer used
log [17:58:27.753] [warning][config][deprecation] Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0."
log [17:58:27.961] [info][plugins-system] Setting up [106] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,translations,licenseApiGuard,legacyExport,embeddable,uiActionsEnhanced,esUiShared,expressions,charts,bfetch,data,home,apmOss,console,consoleExtensions,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,savedObjects,visualizations,visTypeMarkdown,visTypeVislib,visTypeMetric,visTypeVega,visTypeTimelion,features,licenseManagement,watcher,visTypeTable,visTypeTagcloud,visTypeXy,tileMap,regionMap,presentationUtil,canvas,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,indexPatternManagement,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,lens,reporting,lists,encryptedSavedObjects,dataEnhanced,dashboardMode,cloud,upgradeAssistant,snapshotRestore,enterpriseSearch,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,beatsManagement,transform,ingestPipelines,fileUpload,maps,fileDataVisualizer,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,observability,osquery,ml,securitySolution,cases,infra,monitoring,logstash,apm,uptime]
log [17:58:27.963] [info][plugins][taskManager] TaskManager is identified by the Kibana UUID: 05a69651-9eeb-4aaf-96a3-7925f6244ccb
log [17:58:28.172] [warning][config][plugins][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [17:58:28.172] [warning][config][plugins][security] Session cookies will be transmitted over insecure connections. This is not recommended.
log [17:58:28.210] [warning][config][plugins][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [17:58:28.216] [warning][config][plugins][reporting] Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 7.9.2009 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'.
log [17:58:28.217] [warning][encryptedSavedObjects][plugins] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [17:58:28.316] [warning][actions][actions][plugins] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [17:58:28.328] [warning][alerting][alerting][plugins][plugins] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [17:58:28.393] [info][monitoring][monitoring][plugins] config sourced from: production cluster
log [17:58:28.597] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
log [17:58:28.659] [info][savedobjects-service] Starting saved objects migrations
log [17:58:28.699] [info][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH. took: 10ms.
log [17:58:28.715] [info][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH. took: 28ms.
log [17:58:28.720] [info][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH -> UPDATE_TARGET_MAPPINGS. took: 21ms.
log [17:58:28.730] [info][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH -> UPDATE_TARGET_MAPPINGS. took: 15ms.
log [17:58:28.737] [info][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 17ms.
log [17:58:28.788] [info][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 58ms.
log [17:58:29.611] [info][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 823ms.
log [17:58:29.611] [info][savedobjects-service] [.kibana] Migration completed after 924ms
log [17:58:29.646] [info][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 909ms.
log [17:58:29.646] [info][savedobjects-service] [.kibana_task_manager] Migration completed after 957ms
log [17:58:29.711] [info][plugins-system] Starting [106] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,translations,licenseApiGuard,legacyExport,embeddable,uiActionsEnhanced,esUiShared,expressions,charts,bfetch,data,home,apmOss,console,consoleExtensions,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,savedObjects,visualizations,visTypeMarkdown,visTypeVislib,visTypeMetric,visTypeVega,visTypeTimelion,features,licenseManagement,watcher,visTypeTable,visTypeTagcloud,visTypeXy,tileMap,regionMap,presentationUtil,canvas,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,indexPatternManagement,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,lens,reporting,lists,encryptedSavedObjects,dataEnhanced,dashboardMode,cloud,upgradeAssistant,snapshotRestore,enterpriseSearch,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,beatsManagement,transform,ingestPipelines,fileUpload,maps,fileDataVisualizer,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,observability,osquery,ml,securitySolution,cases,infra,monitoring,logstash,apm,uptime]
log [17:58:32.312] [info][server][Kibana][http] http server running at http://10.0.59.155:5601
log [17:58:32.376] [info][kibana-monitoring][monitoring][monitoring][plugins] Starting monitoring stats collection
log [17:58:32.391] [warning][kibana-monitoring][monitoring][monitoring][plugins] ResponseError: export_exception
at onBody (/opt/xplorer/elastic/kibana/kibana-7.13.1-linux-x86_64/node_modules/@elastic/elasticsearch/lib/Transport.js:337:23)
at IncomingMessage.onEnd (/opt/xplorer/elastic/kibana/kibana-7.13.1-linux-x86_64/node_modules/@elastic/elasticsearch/lib/Transport.js:264:11)
at IncomingMessage.emit (events.js:327:22)
at endReadableNT (internal/streams/readable.js:1327:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
log [17:58:32.391] [warning][kibana-monitoring][monitoring][monitoring][plugins] Unable to bulk upload the stats payload to the local cluster
log [17:58:32.397] [info][plugins][securitySolution] Dependent plugin setup complete - Starting ManifestTask
log [17:58:32.976] [info][plugins][reporting] Browser executable: /opt/xplorer/elastic/kibana/kibana-7.13.1-linux-x86_64/x-pack/plugins/reporting/chromium/headless_shell-linux_x64/headless_shell
log [17:58:32.976] [warning][plugins][reporting] Enabling the Chromium sandbox provides an additional layer of protection.
log [17:58:42.387] [warning][kibana-monitoring][monitoring][monitoring][plugins] ResponseError: export_exception
at onBody (/opt/xplorer/elastic/kibana/kibana-7.13.1-linux-x86_64/node_modules/@elastic/elasticsearch/lib/Transport.js:337:23)
at IncomingMessage.onEnd (/opt/xplorer/elastic/kibana/kibana-7.13.1-linux-x86_64/node_modules/@elastic/elasticsearch/lib/Transport.js:264:11)
at IncomingMessage.emit (events.js:327:22)
at endReadableNT (internal/streams/readable.js:1327:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
log [17:58:42.388] [warning][kibana-monitoring][monitoring][monitoring][plugins] Unable to bulk upload the stats payload to the local cluster
log [17:58:52.387] [warning][kibana-monitoring][monitoring][monitoring][plugins] ResponseError: export_exception
at onBody (/opt/xplorer/elastic/kibana/kibana-7.13.1-linux-x86_64/node_modules/@elastic/elasticsearch/lib/Transport.js:337:23)
at IncomingMessage.onEnd (/opt/xplorer/elastic/kibana/kibana-7.13.1-linux-x86_64/node_modules/@elastic/elasticsearch/lib/Transport.js:264:11)
at IncomingMessage.emit (events.js:327:22)
at endReadableNT (internal/streams/readable.js:1327:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
log [17:58:52.387] [warning][kibana-monitoring][monitoring][monitoring][plugins] Unable to bulk upload the stats payload to the local cluster

avnere · November 29, 2021, 12:59pm

elastic Log:

{"type": "server", "timestamp": "2021-11-29T00:00:09,846+02:00", "level": "WARN", "component": "o.e.x.m.MonitoringService", "cluster.name": "elastic-tests-lab162", "node.name": "masterClient-node-1", "message": "monitoring execution failed", "cluster.uuid": "famMKqRTQzmTJbdsxh5ZqA", "node.id": "kYbCqL1dTOK-Vt70n7JFvg" ,
"stacktrace": ["org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks",
"at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$0(ExportBulk.java:110) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:142) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$1(LocalBulk.java:113) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:142) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:38) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:92) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onFailure(ActionListener.java:397) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.bulk.TransportBulkAction.doInternalExecute(TransportBulkAction.java:211) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:162) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:89) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:42) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:149) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:77) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:66) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:484) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:119) [x-pack-core-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:106) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:64) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$1(ExportBulk.java:108) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:103) [x-pack-core-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:124) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:64) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.exporter.Exporters.doExport(Exporters.java:284) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.exporter.Exporters.lambda$export$3(Exporters.java:259) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.exporter.Exporters$AccumulatingExportBulkActionListener.delegateIfComplete(Exporters.java:364) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.exporter.Exporters$AccumulatingExportBulkActionListener.onResponse(Exporters.java:343) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.exporter.Exporters$AccumulatingExportBulkActionListener.onResponse(Exporters.java:314) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.exporter.local.LocalExporter.openBulk(LocalExporter.java:243) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.exporter.Exporters.wrapExportBulk(Exporters.java:245) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:257) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:262) [x-pack-monitoring-7.13.1.jar:7.13.1]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.13.1.jar:7.13.1]",
"at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]",
"at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) [elasticsearch-7.13.1.jar:7.13.1]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
"at java.lang.Thread.run(Thread.java:831) [?:?]",
"Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default_local]",
"... 42 more",
"Caused by: java.lang.IllegalStateException: There are no ingest nodes in this cluster, unable to forward request to an ingest node.",
"at org.elasticsearch.action.ingest.IngestActionForwarder.randomIngestNode(IngestActionForwarder.java:47) ~[elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.ingest.IngestActionForwarder.forwardIngestRequest(IngestActionForwarder.java:40) ~[elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.action.bulk.TransportBulkAction.doInternalExecute(TransportBulkAction.java:208) ~[elasticsearch-7.13.1.jar:7.13.1]",
"... 35 more"] }

avnere · December 1, 2021, 11:42am

Can you help with it?

system · December 29, 2021, 11:42am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.