Cannot generate report in kibana

I am using Elasticsearch 8.7.1 my cluster consists of 2 nodes and the Kibana version is 8.7.1, I am facing a problem with generating CSV it was working normally, but two weeks from now it stopped with the error "reporting task runner has not been initialized!" the full error is below Error: Internal Server Error at fetch_Fetch.fetchResponse (http://x.x.x.x:5601/61224/bundles/core/core.entry.js:1:275461) at async http://x.x.x.x:5601/61224/bundles/core/core.entry.js:1:273517 at async http://x.x.x.x:5601/61224/bundles/core/core.entry.js:1:273474 when I open the reporting page in Kibana I find the report status pending below apart from logging in kibana.log can anyone help

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-11-10T16:18:47.172+02:00","message":"Setting up [132] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,customBranding,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,ftrApis,fieldFormats,expressions,screenshotting,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,cloudDataMigration,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,contentManagement,bfetch,data,watcher,fileUpload,ingestPipelines,ecsDataQualityDashboard,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,eventAnnotation,embeddable,reporting,uiActionsEnhanced,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,aiops,discover,observability,fleet,osquery,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]","log":{"level":"INFO","logger":"plugins-system.standard"},"process":{"pid":1248286},"trace":{"id":"93651a49c2a822f96592f8d01f6540df"},"transaction":{"id":"7513aef5dc31ba56"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-11-10T16:18:48.755+02:00","message":"Starting [132] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,customBranding,usageCollection,taskManager,cloud,guidedOnboarding,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,ftrApis,fieldFormats,expressions,screenshotting,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,cloudDataMigration,advancedSettings,spaces,security,snapshotRestore,lists,encryptedSavedObjects,telemetry,licenseManagement,files,eventLog,actions,notifications,console,contentManagement,bfetch,data,watcher,fileUpload,ingestPipelines,ecsDataQualityDashboard,alerting,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,eventAnnotation,embeddable,reporting,uiActionsEnhanced,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,dataViewFieldEditor,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,aiops,discover,observability,fleet,osquery,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]","log":{"level":"INFO","logger":"plugins-system.standard"},"process":{"pid":1248286},"trace":{"id":"80c1f60306ba40d95b4f5784a98fc653"},"transaction":{"id":"06deb06b50d6cdd2"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-11-10T16:19:04.129+02:00","message":"Reporting task runner has not been initialized!","error":{"message":"Reporting task runner has not been initialized!","type":"Error","stack_trace":"Error: Reporting task runner has not been initialized!\n    at ExecuteReportTask.getTaskManagerStart (/usr/share/kibana/x-pack/plugins/reporting/server/lib/tasks/execute_report.js:113:13)\n    at ExecuteReportTask.scheduleTask (/usr/share/kibana/x-pack/plugins/reporting/server/lib/tasks/execute_report.js:405:23)\n    at ReportingCore.scheduleTask (/usr/share/kibana/x-pack/plugins/reporting/server/core.js:221:35)\n    at RequestHandler.enqueueJob (/usr/share/kibana/x-pack/plugins/reporting/server/routes/lib/request_handler.js:98:34)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at RequestHandler.handleGenerateRequest (/usr/share/kibana/x-pack/plugins/reporting/server/routes/lib/request_handler.js:130:16)\n    at /usr/share/kibana/x-pack/plugins/reporting/server/routes/generate/generate_from_jobparams.js:95:14\n    at Router.handle (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:141:30)\n    at handler (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:107:50)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)"},"log":{"level":"ERROR","logger":"plugins.reporting"},"process":{"pid":1248286},"trace":{"id":"d4d193d1bcfea2639c8ab9182672cf39"},"transaction":{"id":"ff84bdf6896b47fc"}}

Hi @hasan.idriss,

Welcome back! What is the health of your cluster? And what is the size/ record count of the reports you are generating?

I did find an old post from 2022 where I group faced a similar situation and found there was an issue with workload provisioning. The do suggest taking a look at some metrics in that post that could help you identify if that's an issue.

Hope that helps!

1 Like

thank Carly for the fast response
I have 2 Elasticsearch nodes 8core 16 GB RAM 1 tera each
1 fleet server 8core 16 GB
and one Kibana 8core 16 GB RAM with 200 GB HDD
the error on generating any report even with 80 docs

for this post, I tried to scale the Kabana VM according to the mentioned scale doc with no success
when I run curl -X GET "localhost:5601/api/task_manager/_health" -H 'kbn-xsrf: true'
I find that:

"max_throughput_per_minute_per_kibana":200
"avg_required_throughput_per_minute_per_kibana": 101
"result_frequency_percent_as_number":{"Failed":0,"NoAvailableWorkers":0,"NoTasksClaimed":32,"RanOutOfCapacity":38,"RunningAtCapacity":20,"PoolFilled":10},

I can provide the whole result but it is a large file where is the problem and how to solve it, please

It's difficult to say without the whole output, as you've only provided a subset. There is a section in the documentation here that gives some tips on troubleshooting the output that may help.

1 Like
{
  "id": "33d09b4b-03c2-40fb-ba08-fd21ca16504b",
  "timestamp": "2023-11-16T13:43:58.037Z",
  "status": "error",
  "last_update": "2023-11-16T13:43:57.625Z",
  "stats": {
    "configuration": {
      "timestamp": "2023-11-13T07:19:39.737Z",
      "value": {
        "request_capacity": 1000,
        "max_poll_inactivity_cycles": 10,
        "monitored_aggregated_stats_refresh_rate": 60000,
        "monitored_stats_running_average_window": 50,
        "monitored_task_execution_thresholds": {
          "default": {
            "error_threshold": 90,
            "warn_threshold": 80
          },
          "custom": {}
        },
        "poll_interval": 3000,
        "max_workers": 10
      },
      "status": "OK"
    },
    "runtime": {
      "timestamp": "2023-11-16T13:43:57.625Z",
      "value": {
        "polling": {
          "last_successful_poll": "2023-11-16T13:43:57.434Z",
          "last_polling_delay": "2023-11-13T07:19:42.860Z",
          "claim_duration": {
            "p50": 302.5,
            "p90": 1118.5,
            "p95": 2304,
            "p99": 2304
          },
          "duration": {
            "p50": 1093,
            "p90": 2053.5,
            "p95": 2282,
            "p99": 3470
          },
          "claim_conflicts": {
            "p50": 0,
            "p90": 0,
            "p95": 0,
            "p99": 0
          },
          "claim_mismatches": {
            "p50": 0,
            "p90": 0,
            "p95": 0,
            "p99": 0
          },
          "result_frequency_percent_as_number": {
            "Failed": 0,
            "NoAvailableWorkers": 0,
            "NoTasksClaimed": 0,
            "RanOutOfCapacity": 30,
            "RunningAtCapacity": 24,
            "PoolFilled": 46
          },
          "persistence": {
            "recurring": 100,
            "non_recurring": 0
          }
        },
        "drift": {
          "p50": 28259,
          "p90": 31415,
          "p95": 32839,
          "p99": 34252
        },
        "drift_by_type": {
          "reports:monitor": {
            "p50": 56645,
            "p90": 118893.5,
            "p95": 162340,
            "p99": 245422
          },
          "alerting:siem.queryRule": {
            "p50": 21177,
            "p90": 29350,
            "p95": 30523,
            "p99": 31878
          },
          "alerting:siem.eqlRule": {
            "p50": 21352.5,
            "p90": 29157.5,
            "p95": 30952,
            "p99": 34252
          },
          "apm-source-map-migration-task": {
            "p50": 8228,
            "p90": 8228,
            "p95": 8228,
            "p99": 8228
          },
          "alerting:siem.thresholdRule": {
            "p50": 79983.5,
            "p90": 172833,
            "p95": 176252,
            "p99": 180949
          },
          "alerting:monitoring_alert_disk_usage": {
            "p50": 42176,
            "p90": 81715.5,
            "p95": 99695,
            "p99": 181692
          },
          "alerting:monitoring_alert_missing_monitoring_data": {
            "p50": 41675,
            "p90": 80200,
            "p95": 99696,
            "p99": 181692
          },
          "alerting:monitoring_alert_cluster_health": {
            "p50": 41861,
            "p90": 79449.5,
            "p95": 92619,
            "p99": 182513
          },
          "alerting:monitoring_ccr_read_exceptions": {
            "p50": 42696.5,
            "p90": 83287,
            "p95": 110652,
            "p99": 169437
          },
          "alerting:monitoring_alert_nodes_changed": {
            "p50": 40538.5,
            "p90": 82942.5,
            "p95": 113677,
            "p99": 168705
          },
          "endpoint:user-artifact-packager": {
            "p50": 43348,
            "p90": 79291,
            "p95": 100290,
            "p99": 180273
          },
          "alerting:monitoring_alert_cpu_usage": {
            "p50": 45978,
            "p90": 82982.5,
            "p95": 90659,
            "p99": 184003
          },
          "alerting:monitoring_alert_thread_pool_write_rejections": {
            "p50": 43449,
            "p90": 78582,
            "p95": 89652,
            "p99": 182513
          },
          "alerting:siem.newTermsRule": {
            "p50": 77022.5,
            "p90": 181989,
            "p95": 184319,
            "p99": 186773
          },
          "alerting:monitoring_alert_jvm_memory_usage": {
            "p50": 43448.5,
            "p90": 78294,
            "p95": 89653,
            "p99": 182513
          },
          "alerting:monitoring_shard_size": {
            "p50": 46010,
            "p90": 84293,
            "p95": 94563,
            "p99": 184003
          },
          "alerting:monitoring_alert_thread_pool_search_rejections": {
            "p50": 44651,
            "p90": 80221,
            "p95": 101231,
            "p99": 181074
          },
          "dashboard_telemetry": {
            "p50": 11348,
            "p90": 96379,
            "p95": 96379,
            "p99": 96379
          },
          "alerting:siem.indicatorRule": {
            "p50": 4046.5,
            "p90": 50005.5,
            "p95": 125690,
            "p99": 184371
          },
          "security:telemetry-filterlist-artifact": {
            "p50": 5273.5,
            "p90": 33660.5,
            "p95": 56386,
            "p99": 216857
          },
          "alerts_invalidate_api_keys": {
            "p50": 7846,
            "p90": 63874.5,
            "p95": 78319,
            "p99": 177572
          },
          "security:endpoint-diagnostics": {
            "p50": 9060,
            "p90": 76888.5,
            "p95": 81993,
            "p99": 165564
          },
          "cleanup_failed_action_executions": {
            "p50": 8826.5,
            "p90": 50502,
            "p95": 76776,
            "p99": 83604
          },
          "session_cleanup": {
            "p50": 5129,
            "p90": 47498.5,
            "p95": 77598,
            "p99": 102148
          },
          "alerting_health_check": {
            "p50": 7511.5,
            "p90": 32904.5,
            "p95": 76268,
            "p99": 108542
          },
          "Fleet-Usage-Sender": {
            "p50": 7278.5,
            "p90": 30025,
            "p95": 65076,
            "p99": 97703
          },
          "security:telemetry-prebuilt-rule-alerts": {
            "p50": 5771,
            "p90": 35060,
            "p95": 70016,
            "p99": 88266
          },
          "security:telemetry-configuration": {
            "p50": 6545.5,
            "p90": 39168,
            "p95": 70394,
            "p99": 88648
          },
          "Fleet-Usage-Logger": {
            "p50": 3999,
            "p90": 45828,
            "p95": 68966,
            "p99": 168914
          },
          "endpoint:metadata-check-transforms-task": {
            "p50": 3650,
            "p90": 15215.800000000003,
            "p95": 24787.299999999974,
            "p99": 172683
          },
          "security:telemetry-timelines": {
            "p50": 6311.5,
            "p90": 48071.300000000025,
            "p95": 67699.19999999998,
            "p99": 86388
          },
          "alerting:monitoring_alert_kibana_version_mismatch": {
            "p50": 9758,
            "p90": 12980,
            "p95": 12980,
            "p99": 12980
          },
          "alerting:monitoring_alert_logstash_version_mismatch": {
            "p50": 9759,
            "p90": 12980,
            "p95": 12980,
            "p99": 12980
          },
          "osquery:telemetry-configs": {
            "p50": 6121,
            "p90": 14468,
            "p95": 14468,
            "p99": 14468
          },
          "osquery:telemetry-packs": {
            "p50": 5019,
            "p90": 11803,
            "p95": 11803,
            "p99": 11803
          },
          "osquery:telemetry-saved-queries": {
            "p50": 6114,
            "p90": 14165,
            "p95": 14165,
            "p99": 14165
          },
          "alerting:monitoring_alert_license_expiration": {
            "p50": 6114,
            "p90": 14165,
            "p95": 14165,
            "p99": 14165
          },
          "fleet:check-deleted-files-task": {
            "p50": 5741,
            "p90": 13874,
            "p95": 13874,
            "p99": 13874
          },
          "alerting:monitoring_alert_elasticsearch_version_mismatch": {
            "p50": 5524,
            "p90": 13875,
            "p95": 13875,
            "p99": 13875
          },
          "security:endpoint-meta-telemetry": {
            "p50": 9032,
            "p90": 18134,
            "p95": 18134,
            "p99": 18134
          },
          "security:telemetry-detection-rules": {
            "p50": 9727,
            "p90": 14672,
            "p95": 14672,
            "p99": 14672
          },
          "security:telemetry-lists": {
            "p50": 7352,
            "p90": 12791,
            "p95": 12791,
            "p99": 12791
          },
          "apm-telemetry-task": {
            "p50": 9099,
            "p90": 23007.4,
            "p95": 23345,
            "p99": 23345
          },
          "cases-telemetry-task": {
            "p50": 8761,
            "p90": 18535.6,
            "p95": 18906,
            "p99": 18906
          },
          "actions:.server-log": {
            "p50": 59988,
            "p90": 88369,
            "p95": 88369,
            "p99": 88369
          },
          "actions_telemetry": {
            "p50": 9425,
            "p90": 13273,
            "p95": 13273,
            "p99": 13273
          },
          "alerting_telemetry": {
            "p50": 9425,
            "p90": 13273,
            "p95": 13273,
            "p99": 13273
          }
        },
        "load": {
          "p50": 100,
          "p90": 100,
          "p95": 100,
          "p99": 100
        },
        "execution": {
          "duration": {
            "reports:monitor": {
              "p50": 20,
              "p90": 36.5,
              "p95": 42,
              "p99": 710
            },
            "alerting:siem.queryRule": {
              "p50": 1264,
              "p90": 1933,
              "p95": 1991,
              "p99": 2673
            },
            "alerting:siem.eqlRule": {
              "p50": 1447,
              "p90": 2679,
              "p95": 2685,
              "p99": 4060
            },
            "apm-source-map-migration-task": {
              "p50": 16,
              "p90": 16,
              "p95": 16,
              "p99": 16
            },
            "alerting:siem.thresholdRule": {
              "p50": 1218,
              "p90": 2219.5,
              "p95": 2659,
              "p99": 2672
            },
            "alerting:monitoring_alert_disk_usage": {
              "p50": 936.5,
              "p90": 2144.5,
              "p95": 2570,
              "p99": 3610
            },
            "alerting:monitoring_alert_missing_monitoring_data": {
              "p50": 985,
              "p90": 2269.5,
              "p95": 2333,
              "p99": 2811
            },
            "alerting:monitoring_alert_cluster_health": {
              "p50": 885,
              "p90": 1599,
              "p95": 1693,
              "p99": 2814
            },
            "alerting:monitoring_ccr_read_exceptions": {
              "p50": 1116,
              "p90": 2096,
              "p95": 2279,
              "p99": 2381
            },
            "alerting:monitoring_alert_nodes_changed": {
              "p50": 1109,
              "p90": 1932.5,
              "p95": 2603,
              "p99": 3177
            },
            "endpoint:user-artifact-packager": {
              "p50": 1114,
              "p90": 1198,
              "p95": 1272,
              "p99": 1484
            },
            "alerting:monitoring_alert_cpu_usage": {
              "p50": 1041,
              "p90": 1791.5,
              "p95": 1869,
              "p99": 3521
            },
            "alerting:monitoring_alert_thread_pool_write_rejections": {
              "p50": 1135,
              "p90": 1937.5,
              "p95": 2075,
              "p99": 2845
            },
            "alerting:siem.newTermsRule": {
              "p50": 1415.5,
              "p90": 2034,
              "p95": 2586,
              "p99": 4271
            },
            "alerting:monitoring_alert_jvm_memory_usage": {
              "p50": 1095.5,
              "p90": 1736.5,
              "p95": 2064,
              "p99": 3179
            },
            "alerting:monitoring_shard_size": {
              "p50": 1192.5,
              "p90": 1937,
              "p95": 2308,
              "p99": 2674
            },
            "alerting:monitoring_alert_thread_pool_search_rejections": {
              "p50": 1137.5,
              "p90": 2474.5,
              "p95": 3187,
              "p99": 3571
            },
            "dashboard_telemetry": {
              "p50": 452.5,
              "p90": 1881,
              "p95": 1881,
              "p99": 1881
            },
            "alerting:siem.indicatorRule": {
              "p50": 3109.5,
              "p90": 82251.5,
              "p95": 102925,
              "p99": 122757
            },
            "security:telemetry-filterlist-artifact": {
              "p50": 1731.5,
              "p90": 2078,
              "p95": 2165,
              "p99": 5692
            },
            "alerts_invalidate_api_keys": {
              "p50": 17.5,
              "p90": 21,
              "p95": 22,
              "p99": 25
            },
            "security:endpoint-diagnostics": {
              "p50": 969,
              "p90": 1231,
              "p95": 1305,
              "p99": 2123
            },
            "cleanup_failed_action_executions": {
              "p50": 18,
              "p90": 23,
              "p95": 25,
              "p99": 29
            },
            "session_cleanup": {
              "p50": 37,
              "p90": 136.5,
              "p95": 200,
              "p99": 419
            },
            "alerting_health_check": {
              "p50": 128.5,
              "p90": 436.5,
              "p95": 448,
              "p99": 547
            },
            "Fleet-Usage-Sender": {
              "p50": 594,
              "p90": 929,
              "p95": 1161,
              "p99": 1304
            },
            "security:telemetry-prebuilt-rule-alerts": {
              "p50": 10086,
              "p90": 16596,
              "p95": 17476,
              "p99": 22687
            },
            "security:telemetry-configuration": {
              "p50": 1616.5,
              "p90": 1925.5,
              "p95": 2255,
              "p99": 4929
            },
            "Fleet-Usage-Logger": {
              "p50": 172,
              "p90": 221,
              "p95": 238,
              "p99": 359
            },
            "endpoint:metadata-check-transforms-task": {
              "p50": 92,
              "p90": 173,
              "p95": 319.7499999999993,
              "p99": 1291
            },
            "security:telemetry-timelines": {
              "p50": 49947,
              "p90": 165273.20000000007,
              "p95": 183009.2,
              "p99": 189254
            },
            "alerting:monitoring_alert_kibana_version_mismatch": {
              "p50": 1448,
              "p90": 2074,
              "p95": 2074,
              "p99": 2074
            },
            "alerting:monitoring_alert_logstash_version_mismatch": {
              "p50": 995,
              "p90": 1134,
              "p95": 1134,
              "p99": 1134
            },
            "osquery:telemetry-configs": {
              "p50": 19,
              "p90": 58,
              "p95": 58,
              "p99": 58
            },
            "osquery:telemetry-packs": {
              "p50": 101,
              "p90": 139,
              "p95": 139,
              "p99": 139
            },
            "osquery:telemetry-saved-queries": {
              "p50": 170,
              "p90": 222,
              "p95": 222,
              "p99": 222
            },
            "alerting:monitoring_alert_license_expiration": {
              "p50": 562,
              "p90": 736,
              "p95": 736,
              "p99": 736
            },
            "fleet:check-deleted-files-task": {
              "p50": 359,
              "p90": 463,
              "p95": 463,
              "p99": 463
            },
            "alerting:monitoring_alert_elasticsearch_version_mismatch": {
              "p50": 621,
              "p90": 764,
              "p95": 764,
              "p99": 764
            },
            "security:endpoint-meta-telemetry": {
              "p50": 2186,
              "p90": 2786,
              "p95": 2786,
              "p99": 2786
            },
            "security:telemetry-detection-rules": {
              "p50": 9554,
              "p90": 9680,
              "p95": 9680,
              "p99": 9680
            },
            "security:telemetry-lists": {
              "p50": 7936,
              "p90": 8690,
              "p95": 8690,
              "p99": 8690
            },
            "apm-telemetry-task": {
              "p50": 1475,
              "p90": 2098.6,
              "p95": 2140,
              "p99": 2140
            },
            "cases-telemetry-task": {
              "p50": 1747.5,
              "p90": 3811.8000000000006,
              "p95": 3912,
              "p99": 3912
            },
            "actions:.server-log": {
              "p50": 450,
              "p90": 553,
              "p95": 553,
              "p99": 553
            },
            "actions_telemetry": {
              "p50": 10869,
              "p90": 44908,
              "p95": 44908,
              "p99": 44908
            },
            "alerting_telemetry": {
              "p50": 24413,
              "p90": 70170,
              "p95": 70170,
              "p99": 70170
            }
          },
          "duration_by_persistence": {
            "recurring": {
              "p50": 1405.5,
              "p90": 1672.5,
              "p95": 2196,
              "p99": 9840
            },
            "non_recurring": {
              "p50": 1881,
              "p90": 49960.40000000002,
              "p95": 66380.7,
              "p99": 70170
            }
          },

          "persistence": {
            "recurring": 100,
            "non_recurring": 0,
            "ephemeral": 0
          },
          "result_frequency_percent_as_number": {
            "reports:monitor": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:siem.queryRule": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:siem.eqlRule": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "apm-source-map-migration-task": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:siem.thresholdRule": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:monitoring_alert_disk_usage": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:monitoring_alert_missing_monitoring_data": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:monitoring_alert_cluster_health": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:monitoring_ccr_read_exceptions": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:monitoring_alert_nodes_changed": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "endpoint:user-artifact-packager": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:monitoring_alert_cpu_usage": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:monitoring_alert_thread_pool_write_rejections": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:siem.newTermsRule": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:monitoring_alert_jvm_memory_usage": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:monitoring_shard_size": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:monitoring_alert_thread_pool_search_rejections": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "dashboard_telemetry": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:siem.indicatorRule": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "security:telemetry-filterlist-artifact": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerts_invalidate_api_keys": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "security:endpoint-diagnostics": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "cleanup_failed_action_executions": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "session_cleanup": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting_health_check": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "Fleet-Usage-Sender": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "security:telemetry-prebuilt-rule-alerts": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "security:telemetry-configuration": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "Fleet-Usage-Logger": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "endpoint:metadata-check-transforms-task": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "security:telemetry-timelines": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:monitoring_alert_kibana_version_mismatch": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:monitoring_alert_logstash_version_mismatch": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "osquery:telemetry-configs": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "osquery:telemetry-packs": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "osquery:telemetry-saved-queries": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:monitoring_alert_license_expiration": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "fleet:check-deleted-files-task": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting:monitoring_alert_elasticsearch_version_mismatch": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "security:endpoint-meta-telemetry": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "security:telemetry-detection-rules": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "security:telemetry-lists": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "apm-telemetry-task": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "cases-telemetry-task": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "actions:.server-log": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "actions_telemetry": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            },
            "alerting_telemetry": {
              "Success": 100,
              "RetryScheduled": 0,
              "Failed": 0,
              "status": "OK"
            }
          }
        }
      },
      "status": "OK"
    },
    "workload": {
      "timestamp": "2023-11-16T13:43:56.681Z",
      "value": {
        "count": 1467,
        "task_types": {
          "alerting:siem.queryRule": {
            "count": 931,
            "status": {
              "idle": 845,
              "failed": 73,
              "running": 12,
              "claiming": 1
            }
          },
          "alerting:siem.eqlRule": {
            "count": 447,
            "status": {
              "idle": 443,
              "running": 4
            }
          },
          "alerting:siem.thresholdRule": {
            "count": 22,
            "status": {
              "idle": 22
            }
          },
          "alerting:siem.newTermsRule": {
            "count": 21,
            "status": {
              "idle": 20,
              "running": 1
            }
          },
          "alerting:siem.indicatorRule": {
            "count": 6,
            "status": {
              "idle": 6
            }
          },
          "Fleet-Usage-Logger": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "Fleet-Usage-Sender": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "actions_telemetry": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting:monitoring_alert_cluster_health": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting:monitoring_alert_cpu_usage": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting:monitoring_alert_disk_usage": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting:monitoring_alert_elasticsearch_version_mismatch": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting:monitoring_alert_jvm_memory_usage": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting:monitoring_alert_kibana_version_mismatch": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting:monitoring_alert_license_expiration": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting:monitoring_alert_logstash_version_mismatch": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting:monitoring_alert_missing_monitoring_data": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting:monitoring_alert_nodes_changed": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting:monitoring_alert_thread_pool_search_rejections": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting:monitoring_alert_thread_pool_write_rejections": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting:monitoring_ccr_read_exceptions": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting:monitoring_shard_size": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting_health_check": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerting_telemetry": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "alerts_invalidate_api_keys": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "apm-telemetry-task": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "cases-telemetry-task": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "cleanup_failed_action_executions": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "dashboard_telemetry": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "endpoint:metadata-check-transforms-task": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "endpoint:user-artifact-packager": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "fleet:check-deleted-files-task": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "osquery:telemetry-configs": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "osquery:telemetry-packs": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "osquery:telemetry-saved-queries": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "reports:monitor": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:endpoint-diagnostics": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:endpoint-meta-telemetry": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:telemetry-configuration": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:telemetry-detection-rules": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:telemetry-filterlist-artifact": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:telemetry-lists": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:telemetry-prebuilt-rule-alerts": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "security:telemetry-timelines": {
            "count": 1,
            "status": {
              "idle": 1
            }
          },
          "session_cleanup": {
            "count": 1,
            "status": {
              "idle": 1
            }
          }
        },
        "non_recurring": 1467,
        "owner_ids": 1,
        "schedule": [
          [
            "3s",
            1
          ],
          [
            "1m",
            10
          ],
          [
            "60s",
            1
          ],
          [
            "5m",
            1301
          ],
          [
            "10m",
            87
          ],
          [
            "15m",
            2
          ],
          [
            "30m",
            1
          ],
          [
            "45m",
            1
          ],
          [
            "60m",
            32
          ],
          [
            "1h",
            11
          ],
          [
            "3600s",
            1
          ],
          [
            "2h",
            1
          ],
          [
            "3h",
            1
          ],
          [
            "720m",
            2
          ],
          [
            "24h",
            7
          ],
          [
            "1d",
            5
          ]
        ],
        "overdue": 608,
        "overdue_non_recurring": 608,
        "estimated_schedule_density": [
          3,
          9,
          1,
          8,
          10,
          12,
          13,
          3,
          2,
          12,
          9,
          1,
          16,
          11,
          10,
          12,
          11,
          11,
          12,
          11,
          12,
          10,
          9,
          3,
          5,
          7,
          1,
          12,
          10,
          5,
          11,
          2,
          10,
          7,
          9,
          7,
          11,
          17,
          10,
          1
        ],
        "capacity_requirements": {
          "per_minute": 31,
          "per_hour": 16189,
          "per_day": 36
        }
      },
      "status": "OK"
    },
    "capacity_estimation": {
      "status": "error",
      "timestamp": "2023-11-16T13:43:58.037Z",
      "value": {
        "observed": {
          "observed_kibana_instances": 1,
          "max_throughput_per_minute_per_kibana": 200,
          "max_throughput_per_minute": 200,
          "minutes_to_drain_overdue": 608,
          "avg_recurring_required_throughput_per_minute": 301,
          "avg_recurring_required_throughput_per_minute_per_kibana": 301,
          "avg_required_throughput_per_minute": 301,
          "avg_required_throughput_per_minute_per_kibana": 301
        },
        "proposed": {
          "provisioned_kibana": 3,
          "min_required_kibana": 2,
          "avg_recurring_required_throughput_per_minute_per_kibana": 101,
          "avg_required_throughput_per_minute_per_kibana": 101
        }
      }
    }
  }
}

I solved the issue by deploying another Kibana instance
for anyone facing the error above here is my troubleshooting and how I solved it

first I faced the error

Internal Server Error at fetch_Fetch.fetchResponse

when trying to create a CSV report u can see the report in a pending state at stack management-> reporting

so I installed the elastic agent with Kibana integration to monitor Kibana
it states that the Kibana plugin is degraded one of them is the task manager

by using the curl command for the health check

curl -X GET "localhost:5601/api/task_manager/_health" -H 'kbn-xsrf: true'  | json_pp 

note I had added the

| json_pp

to make it jason pretty and readable

in the capacity_estimation-> proposed section it provides the "min_required_kibana" : 2,

for that when I added the new instance issue was solved

hope this helps anyone facing my issue since it was a nightmare for me

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.