Unable to initialize Fleet on Kibana in Ubuntu 22.04

Elastic Stack Kibana
,
1

I'm currently trying to set up a fleet in Kibana as this seems to be a prerequisite for using a suricata module, but I can't get past the error message "Unable to initialize Fleet - An internal server error occured. Check Kibana server logs for details".

I've already looked for answers in this and other forums, but they didn't resolve the issue.

My clusters are all healthy, one node has the "transform"-right and I haven't found any clue in the last 100 loglines that I'll post below.

My kibana.yml-settings are
server.port: 5601
server.host: 127.0.0.1
elasticsearch.host: 127.0.0.1:9300

I've further set the username+password, the ssl.certificateAuthorities as well as the logging-settings for appenders, root and loggers and the pid-file

Here's the log:

[DEBUG][elasticsearch.query.data] 200 - 1.3KB
GET /_xpack
[2023-05-26T22:26:50.581+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":1,"conflicts":"proceed"}
[2023-05-26T22:26:50.601+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":1,"conflicts":"proceed"}
[2023-05-26T22:26:50.621+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":10,"conflicts":"proceed"}
[2023-05-26T22:26:50.859+00:00][DEBUG][elasticsearch.query.data] 200 - 122.0B
GET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip
[2023-05-26T22:26:53.264+00:00][DEBUG][elasticsearch.query.monitoring] 200 - 1.3KB
GET /_xpack
[2023-05-26T22:26:53.360+00:00][DEBUG][elasticsearch.query.data] 200 - 122.0B
GET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip
[2023-05-26T22:26:53.587+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":1,"conflicts":"proceed"}
[2023-05-26T22:26:53.627+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":10,"conflicts":"proceed"}
[2023-05-26T22:26:53.635+00:00][DEBUG][elasticsearch.query.data] 200 - 897.0B
POST /.kibana_task_manager/_search?ignore_unavailable=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"term":{"task.ownerId":"kibana:1803df9f-e99e-48cd-8723-1714141ed9dc"}},{"term":{"task.status":"claiming"}},{"bool":{"should":[{"term":{"task.taskType":"session_cleanup"}},{"term":{"task.taskType":"actions_telemetry"}},{"term":{"task.taskType":"cleanup_failed_action_executions"}},{"term":{"task.taskType":"alerting_telemetry"}},{"term":{"task.taskType":"alerts_invalidate_api_keys"}},{"term":{"task.taskType":"alerting_health_check"}},{"term":{"task.taskType":"reports:monitor"}},{"term":{"task.taskType":"alerting:transform_health"}},{"term":{"task.taskType":"actions:.email"}},{"term":{"task.taskType":"actions:.index"}},{"term":{"task.taskType":"actions:.pagerduty"}},{"term":{"task.taskType":"actions:.swimlane"}},{"term":{"task.taskType":"actions:.server-log"}},{"term":{"task.taskType":"actions:.slack"}},{"term":{"task.taskType":"actions:.webhook"}},{"term":{"task.taskType":"actions:.cases-webhook"}},{"term":{"task.taskType":"actions:.xmatters"}},{"term":{"task.taskType":"actions:.servicenow"}},{"term":{"task.taskType":"actions:.servicenow-sir"}},{"term":{"task.taskType":"actions:.servicenow-itom"}},{"term":{"task.taskType":"actions:.jira"}},{"term":{"task.taskType":"actions:.resilient"}},{"term":{"task.taskType":"actions:.teams"}},{"term":{"task.taskType":"actions:.torq"}},{"term":{"task.taskType":"actions:.opsgenie"}},{"term":{"task.taskType":"actions:.tines"}},{"term":{"task.taskType":"alerting:.index-threshold"}},{"term":{"task.taskType":"alerting:.geo-containment"}},{"term":{"task.taskType":"alerting:.es-query"}},{"term":{"task.taskType":"dashboard_telemetry"}},{"term":{"task.taskType":"cases-telemetry-task"}},{"term":{"task.taskType":"Fleet-Usage-Sender"}},{"term":{"task.taskType":"Fleet-Usage-Logger"}},{"term":{"task.taskType":"fleet:reassign_action:retry"}},{"term":{"task.taskType":"fleet:unenroll_action:retry"}},{"term":{"task.taskType":"fleet:upgrade_action:retry"}},{"term":{"task.taskType":"fleet:update_agent_tags:retry"}},{"term":{"task.taskType":"fleet:request_diagnostics:retry"}},{"term":{"task.taskType":"fleet:check-deleted-files-task"}},{"term":{"task.taskType":"osquery:telemetry-packs"}},{"term":{"task.taskType":"osquery:telemetry-saved-queries"}},{"term":{"task.taskType":"osquery:telemetry-configs"}},{"term":{"task.taskType":"cloud_security_posture-stats_task"}},{"term":{"task.taskType":"ML:saved-objects-sync"}},{"term":{"task.taskType":"alerting:xpack.ml.anomaly_detection_alert"}},{"term":{"task.taskType":"alerting:xpack.ml.anomaly_detection_jobs_health"}},{"term":{"task.taskType":"UPTIME:SyntheticsService:Sync-Saved-Monitor-Objects"}},{"term":{"task.taskType":"alerting:xpack.uptime.alerts.monitorStatus"}},{"term":{"task.taskType":"alerting:xpack.uptime.alerts.tlsCertificate"}},{"term":{"task.taskType":"alerting:xpack.uptime.alerts.durationAnomaly"}},{"term":{"task.taskType":"alerting:xpack.uptime.alerts.tls"}},{"term":{"task.taskType":"alerting:xpack.synthetics.alerts.monitorStatus"}},{"term":{"task.taskType":"alerting:siem.eqlRule"}},{"term":{"task.taskType":"alerting:siem.savedQueryRule"}},{"term":{"task.taskType":"alerting:siem.indicatorRule"}},{"term":{"task.taskType":"alerting:siem.mlRule"}},{"term":{"task.taskType":"alerting:siem.queryRule"}},{"term":{"task.taskType":"alerting:siem.thresholdRule"}},{"term":{"task.taskType":"alerting:siem.newTermsRule"}},{"term":{"task.taskType":"alerting:siem.notifications"}},{"term":{"task.taskType":"endpoint:user-artifact-packager"}},{"term":{"task.taskType":"security:endpoint-diagnostics"}},{"term":{"task.taskType":"security:endpoint-meta-telemetry"}},{"term":{"task.taskType":"security:telemetry-lists"}},{"term":{"task.taskType":"security:telemetry-detection-rules"}},{"term":{"task.taskType":"security:telemetry-prebuilt-rule-alerts"}},{"term":{"task.taskType":"security:telemetry-timelines"}},{"term":{"task.taskType":"security:telemetry-configuration"}},{"term":{"task.taskType":"security:telemetry-filterlist-artifact"}},{"term":{"task.taskType":"endpoint:metadata-check-transforms-task"}},{"term":{"task.taskType":"alerting:metrics.alert.anomaly"}},{"term":{"task.taskType":"alerting:logs.alert.document.count"}},{"term":{"task.taskType":"alerting:metrics.alert.inventory.threshold"}},{"term":{"task.taskType":"alerting:metrics.alert.threshold"}},{"term":{"task.taskType":"alerting:monitoring_alert_cluster_health"}},{"term":{"task.taskType":"alerting:monitoring_alert_license_expiration"}},{"term":{"task.taskType":"alerting:monitoring_alert_cpu_usage"}},{"term":{"task.taskType":"alerting:monitoring_alert_missing_monitoring_data"}},{"term":{"task.taskType":"alerting:monitoring_alert_disk_usage"}},{"term":{"task.taskType":"alerting:monitoring_alert_thread_pool_search_rejections"}},{"term":{"task.taskType":"alerting:monitoring_alert_thread_pool_write_rejections"}},{"term":{"task.taskType":"alerting:monitoring_alert_jvm_memory_usage"}},{"term":{"task.taskType":"alerting:monitoring_alert_nodes_changed"}},{"term":{"task.taskType":"alerting:monitoring_alert_logstash_version_mismatch"}},{"term":{"task.taskType":"alerting:monitoring_alert_kibana_version_mismatch"}},{"term":{"task.taskType":"alerting:monitoring_alert_elasticsearch_version_mismatch"}},{"term":{"task.taskType":"alerting:monitoring_ccr_read_exceptions"}},{"term":{"task.taskType":"alerting:monitoring_shard_size"}},{"term":{"task.taskType":"apm-telemetry-task"}},{"term":{"task.taskType":"alerting:apm.transaction_duration"}},{"term":{"task.taskType":"alerting:apm.anomaly"}},{"term":{"task.taskType":"alerting:apm.error_rate"}},{"term":{"task.taskType":"alerting:apm.transaction_error_rate"}}]}}]}}]}},"size":10,"seq_no_primary_term":true,"sort":{"_script":{"type":"number","order":"asc","script":{"lang":"painless","source":"\nif", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}}
[2023-05-26T22:26:53.649+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":1,"conflicts":"proceed"}
[2023-05-26T22:26:53.690+00:00][DEBUG][elasticsearch.query.data] 200 - 321.0B
POST /_bulk?refresh=false&_source_includes=originId&require_alias=true
{"date": "{"update":{"_id":"task:reports:monitor","_index":".kibana_task_manager_8.7.1","if_seq_no":153973,"if_primary_term":4}}", "_id":"task:reports:monitor","_index":".kibana_task_manager_8.7.1","if_seq_no":153973,"if_primary_term":4}}
{"date": "{"doc":{"task":{"retryAt":"2023-05-26T22:30:53.636Z","runAt":"2023-05-26T22:26:50.617Z","startedAt":"2023-05-26T22:26:53.636Z","params":"{}","ownerId":"kibana:1803df9f-e99e-48cd-8723-1714141ed9dc","schedule":{"interval":"3s"},"taskType":"reports:monitor","traceparent":"00-17c9da0fdbced6e81fae6b30fad68614-880014bf64c1e757-00","state":"{}","scheduledAt":"2023-05-26T22:26:50.617Z","attempts":1,"status":"running"},"updated_at":"2023-05-26T22:26:53.686Z"}}", }","scheduledAt":"2023-05-26T22:26:50.617Z","attempts":1,"status":"running"},"updated_at":"2023-05-26T22:26:53.686Z"}}

[2023-05-26T22:26:53.692+00:00][DEBUG][elasticsearch.query.data] 200 - 159.0B
POST /.reporting-*/_search?_source_excludes=output
{"date": "{"sort":{"created_at":{"order":"asc"}},"query":{"bool":{"filter":{"bool":{"should":[{"bool":{"must":[{"range":{"process_expiration":{"lt":"now"}}},{"terms":{"status":["processing"]}}]}},{"bool":{"must":[{"terms":{"status":["pending"]}}],"must_not":[{"exists":{"field":"migration_version"}}]}}]}}}},"size":1,"seq_no_primary_term":true}", "field":"migration_version"}}]}}]}}}},"size":1,"seq_no_primary_term":true}
[2023-05-26T22:26:53.745+00:00][DEBUG][elasticsearch.query.data] 200 - 321.0B
POST /_bulk?refresh=false&_source_includes=originId&require_alias=true
{"date": "{"update":{"_id":"task:reports:monitor","_index":".kibana_task_manager_8.7.1","if_seq_no":153974,"if_primary_term":4}}", "_id":"task:reports:monitor","_index":".kibana_task_manager_8.7.1","if_seq_no":153974,"if_primary_term":4}}
{"date": "{"doc":{"task":{"runAt":"2023-05-26T22:26:56.636Z","state":"{}","schedule":{"interval":"3s"},"attempts":0,"status":"idle","startedAt":null,"retryAt":null,"ownerId":null,"params":"{}","taskType":"reports:monitor","traceparent":"00-17c9da0fdbced6e81fae6b30fad68614-880014bf64c1e757-00","scheduledAt":"2023-05-26T22:26:50.617Z"},"updated_at":"2023-05-26T22:26:53.742Z"}}", }","taskType":"reports:monitor","traceparent":"00-17c9da0fdbced6e81fae6b30fad68614-880014bf64c1e757-00","scheduledAt":"2023-05-26T22:26:50.617Z"},"updated_at":"2023-05-26T22:26:53.742Z"}}

[2023-05-26T22:26:54.947+00:00][DEBUG][elasticsearch.query.data] 200 - 367.0B
GET /.kibana_8.7.1/_doc/telemetry%3Atelemetry
[2023-05-26T22:26:55.604+00:00][DEBUG][elasticsearch.query.data] 200 - 40.0B
POST /_monitoring/bulk?system_id=kibana&system_api_version=7&interval=10000ms
{"date": "{"index":{"_type":"kibana_stats"}}", "_type":"kibana_stats"}}
{"date": "{"kibana":{"uuid":"1803df9f-e99e-48cd-8723-1714141ed9dc","name":"faui1-213","index":".kibana","host":"127.0.0.1","transport_address":"127.0.0.1:5601","version":"8.7.1","snapshot":false,"status":"green"},"processes":[{"memory":{"heap":{"total_in_bytes":274440192,"used_in_bytes":249888816,"size_limit":2107637760},"resident_set_size_in_bytes":309583872},"pid":72080,"event_loop_delay":10.125678308943089,"event_loop_delay_histogram":{"min":9.05216,"max":16.293887,"mean":10.125678308943089,"exceeds":0,"stddev":0.3823891829871278,"fromTimestamp":"2023-05-26T22:26:48.069Z","lastUpdatedAt":"2023-05-26T22:26:53.061Z","percentiles":{"50":10.125311,"75":10.182655,"95":10.346495,"99":10.575871}},"uptime_in_millis":4462561.608808}],"os":{"platform":"linux","platformRelease":"linux-5.15.0-72-generic","load":{"1m":0.75,"5m":0.54,"15m":0.56},"memory":{"total_in_bytes":4114575360,"free_in_bytes":430526464,"used_in_bytes":3684048896},"uptime_in_millis":206737500,"distro":"Ubuntu","distroRelease":"Ubuntu-22.04","cpu":{"cfs_quota_micros":-1,"cfs_period_micros":100000,"control_group":"/system.slice/kibana.service","stat":{"number_of_elapsed_periods":0,"number_of_times_throttled":0,"time_throttled_nanos":0}},"cpuacct":{"control_group":"/system.slice/kibana.service","usage_nanos":119764352}},"elasticsearch_client":{"totalActiveSockets":0,"totalIdleSockets":3,"totalQueuedRequests":0},"response_times":{"average":0,"max":0},"concurrent_connections":0,"process":{"memory":{"heap":{"total_in_bytes":274440192,"used_in_bytes":249888816,"size_limit":2107637760},"resident_set_size_in_bytes":309583872},"event_loop_delay":10.125678308943089,"event_loop_delay_histogram":{"min":9.05216,"max":16.293887,"mean":10.125678308943089,"exceeds":0,"stddev":0.3823891829871278,"fromTimestamp":"2023-05-26T22:26:48.069Z","lastUpdatedAt":"2023-05-26T22:26:53.061Z","percentiles":{"50":10.125311,"75":10.182655,"95":10.346495,"99":10.575871}},"uptime_in_millis":4462561.608808},"requests":{"disconnects":0,"total":0},"timestamp":"2023-05-26T22:26:53.067Z"}", "disconnects":0,"total":0},"timestamp":"2023-05-26T22:26:53.067Z"}
{"date": "{"index":{"_type":"kibana_settings"}}", "_type":"kibana_settings"}}
{"date": "{"kibana":{"uuid":"1803df9f-e99e-48cd-8723-1714141ed9dc","name":"faui1-213","index":".kibana","host":"127.0.0.1","locale":"en","port":"5601","transport_address":"127.0.0.1:5601","version":"8.7.1","snapshot":false,"status":"green"}}", "uuid":"1803df9f-e99e-48cd-8723-1714141ed9dc","name":"faui1-213","index":".kibana","host":"127.0.0.1","locale":"en","port":"5601","transport_address":"127.0.0.1:5601","version":"8.7.1","snapshot":false,"status":"green"}}

[2023-05-26T22:26:55.859+00:00][DEBUG][elasticsearch.query.data] 200 - 122.0B
GET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip
[2023-05-26T22:26:56.577+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":10,"conflicts":"proceed"}
[2023-05-26T22:26:56.601+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":1,"conflicts":"proceed"}
[2023-05-26T22:26:56.618+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":1,"conflicts":"proceed"}
[2023-05-26T22:26:58.359+00:00][DEBUG][elasticsearch.query.data] 200 - 122.0B
GET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip
[2023-05-26T22:26:59.588+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":1,"conflicts":"proceed"}
[2023-05-26T22:26:59.620+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":1,"conflicts":"proceed"}
[2023-05-26T22:26:59.677+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":10,"conflicts":"proceed"}
[2023-05-26T22:26:59.686+00:00][DEBUG][elasticsearch.query.data] 200 - 897.0B
POST /.kibana_task_manager/_search?ignore_unavailable=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"term":{"task.ownerId":"kibana:1803df9f-e99e-48cd-8723-1714141ed9dc"}},{"term":{"task.status":"claiming"}},{"bool":{"should":[{"term":{"task.taskType":"session_cleanup"}},{"term":{"task.taskType":"actions_telemetry"}},{"term":{"task.taskType":"cleanup_failed_action_executions"}},{"term":{"task.taskType":"alerting_telemetry"}},{"term":{"task.taskType":"alerts_invalidate_api_keys"}},{"term":{"task.taskType":"alerting_health_check"}},{"term":{"task.taskType":"reports:monitor"}},{"term":{"task.taskType":"alerting:transform_health"}},{"term":{"task.taskType":"actions:.email"}},{"term":{"task.taskType":"actions:.index"}},{"term":{"task.taskType":"actions:.pagerduty"}},{"term":{"task.taskType":"actions:.swimlane"}},{"term":{"task.taskType":"actions:.server-log"}},{"term":{"task.taskType":"actions:.slack"}},{"term":{"task.taskType":"actions:.webhook"}},{"term":{"task.taskType":"actions:.cases-webhook"}},{"term":{"task.taskType":"actions:.xmatters"}},{"term":{"task.taskType":"actions:.servicenow"}},{"term":{"task.taskType":"actions:.servicenow-sir"}},{"term":{"task.taskType":"actions:.servicenow-itom"}},{"term":{"task.taskType":"actions:.jira"}},{"term":{"task.taskType":"actions:.resilient"}},{"term":{"task.taskType":"actions:.teams"}},{"term":{"task.taskType":"actions:.torq"}},{"term":{"task.taskType":"actions:.opsgenie"}},{"term":{"task.taskType":"actions:.tines"}},{"term":{"task.taskType":"alerting:.index-threshold"}},{"term":{"task.taskType":"alerting:.geo-containment"}},{"term":{"task.taskType":"alerting:.es-query"}},{"term":{"task.taskType":"dashboard_telemetry"}},{"term":{"task.taskType":"cases-telemetry-task"}},{"term":{"task.taskType":"Fleet-Usage-Sender"}},{"term":{"task.taskType":"Fleet-Usage-Logger"}},{"term":{"task.taskType":"fleet:reassign_action:retry"}},{"term":{"task.taskType":"fleet:unenroll_action:retry"}},{"term":{"task.taskType":"fleet:upgrade_action:retry"}},{"term":{"task.taskType":"fleet:update_agent_tags:retry"}},{"term":{"task.taskType":"fleet:request_diagnostics:retry"}},{"term":{"task.taskType":"fleet:check-deleted-files-task"}},{"term":{"task.taskType":"osquery:telemetry-packs"}},{"term":{"task.taskType":"osquery:telemetry-saved-queries"}},{"term":{"task.taskType":"osquery:telemetry-configs"}},{"term":{"task.taskType":"cloud_security_posture-stats_task"}},{"term":{"task.taskType":"ML:saved-objects-sync"}},{"term":{"task.taskType":"alerting:xpack.ml.anomaly_detection_alert"}},{"term":{"task.taskType":"alerting:xpack.ml.anomaly_detection_jobs_health"}},{"term":{"task.taskType":"UPTIME:SyntheticsService:Sync-Saved-Monitor-Objects"}},{"term":{"task.taskType":"alerting:xpack.uptime.alerts.monitorStatus"}},{"term":{"task.taskType":"alerting:xpack.uptime.alerts.tlsCertificate"}},{"term":{"task.taskType":"alerting:xpack.uptime.alerts.durationAnomaly"}},{"term":{"task.taskType":"alerting:xpack.uptime.alerts.tls"}},{"term":{"task.taskType":"alerting:xpack.synthetics.alerts.monitorStatus"}},{"term":{"task.taskType":"alerting:siem.eqlRule"}},{"term":{"task.taskType":"alerting:siem.savedQueryRule"}},{"term":{"task.taskType":"alerting:siem.indicatorRule"}},{"term":{"task.taskType":"alerting:siem.mlRule"}},{"term":{"task.taskType":"alerting:siem.queryRule"}},{"term":{"task.taskType":"alerting:siem.thresholdRule"}},{"term":{"task.taskType":"alerting:siem.newTermsRule"}},{"term":{"task.taskType":"alerting:siem.notifications"}},{"term":{"task.taskType":"endpoint:user-artifact-packager"}},{"term":{"task.taskType":"security:endpoint-diagnostics"}},{"term":{"task.taskType":"security:endpoint-meta-telemetry"}},{"term":{"task.taskType":"security:telemetry-lists"}},{"term":{"task.taskType":"security:telemetry-detection-rules"}},{"term":{"task.taskType":"security:telemetry-prebuilt-rule-alerts"}},{"term":{"task.taskType":"security:telemetry-timelines"}},{"term":{"task.taskType":"security:telemetry-configuration"}},{"term":{"task.taskType":"security:telemetry-filterlist-artifact"}},{"term":{"task.taskType":"endpoint:metadata-check-transforms-task"}},{"term":{"task.taskType":"alerting:metrics.alert.anomaly"}},{"term":{"task.taskType":"alerting:logs.alert.document.count"}},{"term":{"task.taskType":"alerting:metrics.alert.inventory.threshold"}},{"term":{"task.taskType":"alerting:metrics.alert.threshold"}},{"term":{"task.taskType":"alerting:monitoring_alert_cluster_health"}},{"term":{"task.taskType":"alerting:monitoring_alert_license_expiration"}},{"term":{"task.taskType":"alerting:monitoring_alert_cpu_usage"}},{"term":{"task.taskType":"alerting:monitoring_alert_missing_monitoring_data"}},{"term":{"task.taskType":"alerting:monitoring_alert_disk_usage"}},{"term":{"task.taskType":"alerting:monitoring_alert_thread_pool_search_rejections"}},{"term":{"task.taskType":"alerting:monitoring_alert_thread_pool_write_rejections"}},{"term":{"task.taskType":"alerting:monitoring_alert_jvm_memory_usage"}},{"term":{"task.taskType":"alerting:monitoring_alert_nodes_changed"}},{"term":{"task.taskType":"alerting:monitoring_alert_logstash_version_mismatch"}},{"term":{"task.taskType":"alerting:monitoring_alert_kibana_version_mismatch"}},{"term":{"task.taskType":"alerting:monitoring_alert_elasticsearch_version_mismatch"}},{"term":{"task.taskType":"alerting:monitoring_ccr_read_exceptions"}},{"term":{"task.taskType":"alerting:monitoring_shard_size"}},{"term":{"task.taskType":"apm-telemetry-task"}},{"term":{"task.taskType":"alerting:apm.transaction_duration"}},{"term":{"task.taskType":"alerting:apm.anomaly"}},{"term":{"task.taskType":"alerting:apm.error_rate"}},{"term":{"task.taskType":"alerting:apm.transaction_error_rate"}}]}}]}}]}},"size":10,"seq_no_primary_term":true,"sort":{"_script":{"type":"number","order":"asc","script":{"lang":"painless","source":"\nif", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}}
[2023-05-26T22:26:59.742+00:00][DEBUG][elasticsearch.query.data] 200 - 321.0B
POST /_bulk?refresh=false&_source_includes=originId&require_alias=true
{"date": "{"update":{"_id":"task:reports:monitor","_index":".kibana_task_manager_8.7.1","if_seq_no":153976,"if_primary_term":4}}", "_id":"task:reports:monitor","_index":".kibana_task_manager_8.7.1","if_seq_no":153976,"if_primary_term":4}}
{"date": "{"doc":{"task":{"retryAt":"2023-05-26T22:30:59.687Z","runAt":"2023-05-26T22:26:56.636Z","startedAt":"2023-05-26T22:26:59.687Z","params":"{}","ownerId":"kibana:1803df9f-e99e-48cd-8723-1714141ed9dc","schedule":{"interval":"3s"},"taskType":"reports:monitor","traceparent":"00-17c9da0fdbced6e81fae6b30fad68614-880014bf64c1e757-00","state":"{}","scheduledAt":"2023-05-26T22:26:56.636Z","attempts":1,"status":"running"},"updated_at":"2023-05-26T22:26:59.737Z"}}", }","scheduledAt":"2023-05-26T22:26:56.636Z","attempts":1,"status":"running"},"updated_at":"2023-05-26T22:26:59.737Z"}}

[2023-05-26T22:26:59.748+00:00][DEBUG][elasticsearch.query.data] 200 - 159.0B
POST /.reporting-*/_search?_source_excludes=output
{"date": "{"sort":{"created_at":{"order":"asc"}},"query":{"bool":{"filter":{"bool":{"should":[{"bool":{"must":[{"range":{"process_expiration":{"lt":"now"}}},{"terms":{"status":["processing"]}}]}},{"bool":{"must":[{"terms":{"status":["pending"]}}],"must_not":[{"exists":{"field":"migration_version"}}]}}]}}}},"size":1,"seq_no_primary_term":true}", "field":"migration_version"}}]}}]}}}},"size":1,"seq_no_primary_term":true}
[2023-05-26T22:26:59.806+00:00][DEBUG][elasticsearch.query.data] 200 - 321.0B
POST /_bulk?refresh=false&_source_includes=originId&require_alias=true
{"date": "{"update":{"_id":"task:reports:monitor","_index":".kibana_task_manager_8.7.1","if_seq_no":153977,"if_primary_term":4}}", "_id":"task:reports:monitor","_index":".kibana_task_manager_8.7.1","if_seq_no":153977,"if_primary_term":4}}
{"date": "{"doc":{"task":{"runAt":"2023-05-26T22:27:02.687Z","state":"{}","schedule":{"interval":"3s"},"attempts":0,"status":"idle","startedAt":null,"retryAt":null,"ownerId":null,"params":"{}","taskType":"reports:monitor","traceparent":"00-17c9da0fdbced6e81fae6b30fad68614-880014bf64c1e757-00","scheduledAt":"2023-05-26T22:26:56.636Z"},"updated_at":"2023-05-26T22:26:59.799Z"}}", }","taskType":"reports:monitor","traceparent":"00-17c9da0fdbced6e81fae6b30fad68614-880014bf64c1e757-00","scheduledAt":"2023-05-26T22:26:56.636Z"},"updated_at":"2023-05-26T22:26:59.799Z"}}

[2023-05-26T22:27:00.859+00:00][DEBUG][elasticsearch.query.data] 200 - 122.0B
GET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip
[2023-05-26T22:27:02.578+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":1,"conflicts":"proceed"}
[2023-05-26T22:27:02.606+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":10,"conflicts":"proceed"}
[2023-05-26T22:27:02.624+00:00][DEBUG][elasticsearch.query.data] 200 - 228.0B
POST /.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true
{"date": "{"query":{"bool":{"must":[{"term":{"type":"task"}},{"bool":{"must":[{"bool":{"must":[{"term":{"task.enabled":true}}]}},{"bool":{"should":[{"bool":{"must":[{"term":{"task.status":"idle"}},{"range":{"task.runAt":{"lte":"now"}}}]}},{"bool":{"must":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}]}},{"range":{"task.retryAt":{"lte":"now"}}}]}}]}}],"filter":[{"bool":{"must_not":[{"bool":{"should":[{"term":{"task.status":"running"}},{"term":{"task.status":"claiming"}}],"must":{"range":{"task.retryAt":{"gt":"now"}}}}}]}}]}}]}},"script":{"source":"\n", \n  return doc['task.runAt'].value.toInstant().toEpochMilli();\n}\n    "}}}],"max_docs":1,"conflicts":"proceed"}

I'd very much appreciate any help or clues on how to resolve this issue.

2

Welcome to our community! :smiley:

Is that from /var/log/kibana/kibana/log?

That should be 127.0.0.1:9200 as Kibana uses the API and not the transport protocol.

3

Hi, thank you for your reply.

Is that from /var/log/kibana/kibana/log?

Yes, respectively from /var/log/kibana/kibana.log.

That should be 127.0.0.1:9200 as Kibana uses the API and not the transport protocol.

That distinction really confuses me. I'm not sure whether I understood the answer in this thread correctly, but 9300 seemed to be the obligatory port to set and 9200 only an optional one. But maybe I got it the wrong way around.
Now that I've set the port to 9200 in the kibana.yml, the "systemctl status kibana" command reveals a running kibana service but also an error ECONNREFUSED. When I test the connection with curl, the command with port 9300 returns a valid answer, but the command with port 9200 only returns a "connection refused" error. How do I proceed from here to get the connection through?

4

Could it be an issue with the RAM?
free --mega shows 210 free MB on the Mem and 304 free MB on the Swap.

ETA: I forgot to edit the port in the elasticsearch.yml accordingly.
Now I at least see the login screen but the login fails due to missing writing space on the device.

5

ETA: I cleared the space, am able to login again but I still get the same error message regarding fleet initialisation and the Suricata Event Dashboard also doesn't show any events or activities. Is there something more to connect for this to work?

6

Update: Okay, there seem to be some issues with the logstash-configuration. I'll try to construct the complete elk-stack appropriately and open a new issue in case the issue remains unresolved.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.