Kibana 8.11.0 Failed To Start (Exit Code 1)

Elastic Stack Kibana
1

I am running a ELK Stack in a local hosted Docker Container that comprises of sub containers of Elasticsearch, Kibana and Logstash service.

image
image1362×389 52.1 KB

Setup / config wise, I followed Ali Younges youtube video closely to setup.

https://youtu.be/jXU_1GADENQ?si=VVSQBkNpeyM8iVpO

I have a Kibana <-> Elastic search connection issue
The supposed index cannot be found? Im confused.
I get timed out
Unable to connect to Elasticsearch. Error: index_not_found_exception
Kibana is reporting that it cannot authenticate due to the absence of a license.

I am not sure whats going on. This setup has to work given that its by a reliable YouTuber...

Pls help me. And tell me what's wrong and how I can fix it.

Docker-Compose.yml

kibana:
depends_on:
es01:
condition: service_healthy
# es02:
# condition: service_healthy
# es03:
# condition: service_healthy
image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
labels:
co.elastic.logs/module: kibana
volumes:
- certs:/usr/share/kibana/config/certs
- kibanadata:/usr/share/kibana/data
ports:
- ${KIBANA_PORT}:5601
environment:
- SERVERNAME=kibana
- ELASTICSEARCH_HOSTS=https://es01:9200
- ELASTICSEARCH_USERNAME=kibana_system
- ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
- ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
mem_limit: ${KB_MEM_LIMIT}
healthcheck:
test:
[
"CMD-SHELL",
"curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
]
interval: 10s
timeout: 10s
retries: 120

Kibana Logs

2023-11-11 02:14:20 [2023-11-10T18:14:20.442+00:00][WARN ][savedobjects-service] Unable to connect to Elasticsearch. Error: index_not_found_exception
2023-11-11 02:14:20 Root causes:
2023-11-11 02:14:20 index_not_found_exception: no such index [.kibana]
2023-11-11 02:14:33 [2023-11-10T18:14:33.603+00:00][INFO ][savedobjects-service] [.kibana_analytics] INIT -> CREATE_NEW_TARGET. took: 12409ms.
2023-11-11 02:14:34 [2023-11-10T18:14:34.593+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] INIT -> CREATE_NEW_TARGET. took: 12073ms.
2023-11-11 02:14:34 [2023-11-10T18:14:34.650+00:00][INFO ][savedobjects-service] [.kibana_security_solution] INIT -> CREATE_NEW_TARGET. took: 12534ms.
2023-11-11 02:14:34 [2023-11-10T18:14:34.786+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> CREATE_NEW_TARGET. took: 13685ms.
2023-11-11 02:14:34 [2023-11-10T18:14:34.863+00:00][INFO ][savedobjects-service] [.kibana_ingest] INIT -> CREATE_NEW_TARGET. took: 11845ms.
2023-11-11 02:14:34 [2023-11-10T18:14:34.914+00:00][INFO ][savedobjects-service] [.kibana] INIT -> CREATE_NEW_TARGET. took: 14054ms.
2023-11-11 02:15:07 [2023-11-10T18:15:06.831+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'process_cluster_event_timeout_exception
2023-11-11 02:15:07 Root causes:
2023-11-11 02:15:07 process_cluster_event_timeout_exception: failed to process cluster event (create-index [.kibana_8.11.0_001], cause [api]) within 30s'. Retrying attempt 1 in 2 seconds.
2023-11-11 02:15:07 [2023-11-10T18:15:07.195+00:00][INFO ][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 31895ms.
2023-11-11 02:15:07 [2023-11-10T18:15:07.603+00:00][ERROR][savedobjects-service] [.kibana_alerting_cases] Action failed with 'process_cluster_event_timeout_exception
2023-11-11 02:15:07 Root causes:
2023-11-11 02:15:07 process_cluster_event_timeout_exception: failed to process cluster event (create-index [.kibana_alerting_cases_8.11.0_001], cause [api]) within 30s'. Retrying attempt 1 in 2 seconds.
2023-11-11 02:15:07 [2023-11-10T18:15:07.606+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 33010ms.
2023-11-11 02:15:07 [2023-11-10T18:15:07.639+00:00][ERROR][savedobjects-service] [.kibana_security_solution] Action failed with 'process_cluster_event_timeout_exception
2023-11-11 02:15:07 Root causes:
2023-11-11 02:15:07 process_cluster_event_timeout_exception: failed to process cluster event (create-index [.kibana_security_solution_8.11.0_001], cause [api]) within 30s'. Retrying attempt 1 in 2 seconds.
2023-11-11 02:15:07 [2023-11-10T18:15:07.642+00:00][INFO ][savedobjects-service] [.kibana_security_solution] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 32991ms.
2023-11-11 02:15:24 [2023-11-10T18:15:24.439+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell
2023-11-11 02:16:44 [2023-11-10T18:16:44.676+00:00][ERROR][savedobjects-service] [.kibana_analytics] Action failed with 'Request timed out'. Retrying attempt 1 in 2 seconds.
2023-11-11 02:16:44 [2023-11-10T18:16:44.825+00:00][INFO ][savedobjects-service] [.kibana_analytics] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 131093ms.
2023-11-11 02:16:44 [2023-11-10T18:16:44.934+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'Request timed out'. Retrying attempt 1 in 2 seconds.
2023-11-11 02:16:44 [2023-11-10T18:16:44.943+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 130149ms.
2023-11-11 02:17:01 [2023-11-10T18:17:01.360+00:00][ERROR][savedobjects-service] [.kibana_ingest] Action failed with 'Request timed out'. Retrying attempt 1 in 2 seconds.
2023-11-11 02:17:01 [2023-11-10T18:17:01.682+00:00][INFO ][savedobjects-service] [.kibana_ingest] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 146493ms.
2023-11-11 02:17:02 [2023-11-10T18:17:02.834+00:00][ERROR][savedobjects-service] [.kibana_alerting_cases] Action failed with 'process_cluster_event_timeout_exception
2023-11-11 02:17:02 Root causes:
2023-11-11 02:17:02 process_cluster_event_timeout_exception: failed to process cluster event (create-index [.kibana_alerting_cases_8.11.0_001], cause [api]) within 30s'. Retrying attempt 2 in 4 seconds.
2023-11-11 02:17:02 [2023-11-10T18:17:02.896+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 115228ms.
2023-11-11 02:17:03 [2023-11-10T18:17:02.992+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'process_cluster_event_timeout_exception
2023-11-11 02:17:03 Root causes:
2023-11-11 02:17:03 process_cluster_event_timeout_exception: failed to process cluster event (create-index [.kibana_8.11.0_001], cause [api]) within 30s'. Retrying attempt 2 in 4 seconds.
2023-11-11 02:17:03 [2023-11-10T18:17:02.998+00:00][INFO ][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 116184ms.
2023-11-11 02:20:44 [2023-11-10T18:20:44.523+00:00][ERROR][savedobjects-service] [.kibana_security_solution] Action failed with 'Request timed out'. Retrying attempt 2 in 4 seconds.
2023-11-11 02:20:44 [2023-11-10T18:20:44.642+00:00][INFO ][savedobjects-service] [.kibana_security_solution] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 336882ms.
2023-11-11 02:20:44 [2023-11-10T18:20:44.720+00:00][ERROR][savedobjects-service] [.kibana_ingest] Action failed with 'Request timed out'. Retrying attempt 2 in 4 seconds.
2023-11-11 02:20:44 [2023-11-10T18:20:44.722+00:00][INFO ][savedobjects-service] [.kibana_ingest] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 223363ms.
2023-11-11 02:21:10 [2023-11-10T18:21:10.019+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 265057ms.
2023-11-11 02:21:13 [2023-11-10T18:21:12.981+00:00][INFO ][savedobjects-service] [.kibana_analytics] CREATE_NEW_TARGET -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 268299ms.
2023-11-11 02:23:30 [2023-11-10T18:23:30.017+00:00][ERROR][savedobjects-service] [.kibana_ingest] Action failed with 'Request timed out'. Retrying attempt 3 in 8 seconds.
2023-11-11 02:23:30 [2023-11-10T18:23:30.189+00:00][INFO ][savedobjects-service] [.kibana_ingest] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 165295ms.
2023-11-11 02:23:30 [2023-11-10T18:23:30.313+00:00][ERROR][savedobjects-service] [.kibana_security_solution] Action failed with 'Request timed out'. Retrying attempt 3 in 8 seconds.
2023-11-11 02:23:30 [2023-11-10T18:23:30.315+00:00][INFO ][savedobjects-service] [.kibana_security_solution] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 165792ms.
2023-11-11 02:23:31 [2023-11-10T18:23:31.308+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 141316ms.
2023-11-11 02:23:31 [2023-11-10T18:23:31.975+00:00][INFO ][savedobjects-service] [.kibana_analytics] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 139004ms.
2023-11-11 02:23:32 [2023-11-10T18:23:32.285+00:00][ERROR][savedobjects-service] [.kibana_alerting_cases] Action failed with 'Client network socket disconnected before secure TLS connection was established - Local: unknown:unknown, Remote: unknown:unknown'. Retrying attempt 3 in 8 seconds.
2023-11-11 02:23:32 [2023-11-10T18:23:32.291+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 389454ms.
2023-11-11 02:23:32 [2023-11-10T18:23:32.324+00:00][ERROR][savedobjects-service] [.kibana] Action failed with 'Client network socket disconnected before secure TLS connection was established - Local: unknown:unknown, Remote: unknown:unknown'. Retrying attempt 3 in 8 seconds.
2023-11-11 02:23:32 [2023-11-10T18:23:32.326+00:00][INFO ][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 389332ms.
2023-11-11 02:23:50 [2023-11-10T18:23:50.239+00:00][INFO ][savedobjects-service] [.kibana_analytics] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 18261ms.
2023-11-11 02:23:50 [2023-11-10T18:23:50.385+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 19078ms.
2023-11-11 02:23:50 [2023-11-10T18:23:50.543+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. socket hang up - Local: 172.18.0.6:35956, Remote: unknown:unknown
2023-11-11 02:23:51 [2023-11-10T18:23:51.612+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 1227ms.
2023-11-11 02:23:51 [2023-11-10T18:23:51.652+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_PROPERTIES. took: 40ms.
2023-11-11 02:23:51 [2023-11-10T18:23:51.742+00:00][INFO ][savedobjects-service] [.kibana_analytics] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 1506ms.
2023-11-11 02:23:51 [2023-11-10T18:23:51.748+00:00][INFO ][savedobjects-service] [.kibana_analytics] CHECK_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_PROPERTIES. took: 7ms.
2023-11-11 02:24:11 [2023-11-10T18:24:11.186+00:00][INFO ][savedobjects-service] [.kibana_security_solution] CREATE_NEW_TARGET -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 40866ms.
2023-11-11 02:24:11 [2023-11-10T18:24:11.446+00:00][INFO ][savedobjects-service] [.kibana_ingest] CREATE_NEW_TARGET -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 41432ms.
2023-11-11 02:24:12 [2023-11-10T18:24:12.877+00:00][INFO ][savedobjects-service] [.kibana_security_solution] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 1698ms.
2023-11-11 02:24:14 [2023-11-10T18:24:14.615+00:00][INFO ][savedobjects-service] [.kibana_ingest] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 3169ms.
2023-11-11 02:24:29 [2023-11-10T18:24:28.682+00:00][ERROR][savedobjects-service] [.kibana_analytics] Action failed with 'process_cluster_event_timeout_exception
2023-11-11 02:24:29 Root causes:
2023-11-11 02:24:29 process_cluster_event_timeout_exception: failed to process cluster event (put-mapping [.kibana_analytics_8.11.0_001/hMcfir4ATKaUsVtsXotqbA]) within 30s'. Retrying attempt 1 in 2 seconds.
2023-11-11 02:24:29 [2023-11-10T18:24:29.137+00:00][INFO ][savedobjects-service] [.kibana_analytics] UPDATE_TARGET_MAPPINGS_PROPERTIES -> UPDATE_TARGET_MAPPINGS_PROPERTIES. took: 36923ms.
2023-11-11 02:24:29 [2023-11-10T18:24:29.979+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'process_cluster_event_timeout_exception
2023-11-11 02:24:29 Root causes:
2023-11-11 02:24:29 process_cluster_event_timeout_exception: failed to process cluster event (put-mapping [.kibana_task_manager_8.11.0_001/-5pVGH3nRXOzKyEQSn9OHw]) within 30s'. Retrying attempt 1 in 2 seconds.
2023-11-11 02:24:29 [2023-11-10T18:24:29.982+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS_PROPERTIES -> UPDATE_TARGET_MAPPINGS_PROPERTIES. took: 38327ms.
2023-11-11 02:24:31 [2023-11-10T18:24:31.162+00:00][INFO ][savedobjects-service] [.kibana_security_solution] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 18285ms.
2023-11-11 02:24:31 [2023-11-10T18:24:31.379+00:00][INFO ][savedobjects-service] [.kibana_ingest] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 16764ms.
2023-11-11 02:24:38 [2023-11-10T18:24:37.747+00:00][INFO ][savedobjects-service] [.kibana_security_solution] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 6570ms.
2023-11-11 02:24:38 [2023-11-10T18:24:38.325+00:00][INFO ][savedobjects-service] [.kibana_security_solution] CHECK_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_PROPERTIES. took: 593ms.
2023-11-11 02:24:39 [2023-11-10T18:24:39.077+00:00][INFO ][savedobjects-service] [.kibana_ingest] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 7698ms.
2023-11-11 02:24:39 [2023-11-10T18:24:39.116+00:00][INFO ][savedobjects-service] [.kibana_ingest] CHECK_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_PROPERTIES. took: 39ms.
2023-11-11 02:24:43 [2023-11-10T18:24:43.329+00:00][INFO ][savedobjects-service] [.kibana_analytics] UPDATE_TARGET_MAPPINGS_PROPERTIES -> UPDATE_TARGET_MAPPINGS_PROPERTIES_WAIT_FOR_TASK. took: 14643ms.
2023-11-11 02:24:43 [2023-11-10T18:24:43.622+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS_PROPERTIES -> UPDATE_TARGET_MAPPINGS_PROPERTIES_WAIT_FOR_TASK. took: 13642ms.
2023-11-11 02:24:44 [2023-11-10T18:24:44.045+00:00][INFO ][savedobjects-service] [.kibana_security_solution] UPDATE_TARGET_MAPPINGS_PROPERTIES -> UPDATE_TARGET_MAPPINGS_PROPERTIES_WAIT_FOR_TASK. took: 5720ms.
2023-11-11 02:24:45 [2023-11-10T18:24:45.595+00:00][INFO ][savedobjects-service] [.kibana_ingest] UPDATE_TARGET_MAPPINGS_PROPERTIES -> UPDATE_TARGET_MAPPINGS_PROPERTIES_WAIT_FOR_TASK. took: 6479ms.
2023-11-11 02:24:59 [2023-11-10T18:24:59.285+00:00][INFO ][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 86951ms.
2023-11-11 02:24:59 [2023-11-10T18:24:59.448+00:00][INFO ][savedobjects-service] [.kibana] CHECK_VERSION_INDEX_READY_ACTIONS -> MARK_VERSION_INDEX_READY. took: 173ms.
2023-11-11 02:24:59 [2023-11-10T18:24:59.613+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] CREATE_NEW_TARGET -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 87327ms.
2023-11-11 02:24:59 [2023-11-10T18:24:59.623+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] CHECK_VERSION_INDEX_READY_ACTIONS -> MARK_VERSION_INDEX_READY. took: 11ms.
2023-11-11 02:25:03 [2023-11-10T18:25:03.266+00:00][INFO ][savedobjects-service] [.kibana] MARK_VERSION_INDEX_READY -> DONE. took: 3812ms.
2023-11-11 02:25:03 [2023-11-10T18:25:03.331+00:00][INFO ][savedobjects-service] [.kibana] Migration completed after 642464ms
2023-11-11 02:25:03 [2023-11-10T18:25:03.562+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] MARK_VERSION_INDEX_READY -> DONE. took: 3939ms.
2023-11-11 02:25:03 [2023-11-10T18:25:03.566+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] Migration completed after 641045ms
2023-11-11 02:25:09 [2023-11-10T18:25:09.481+00:00][INFO ][savedobjects-service] [.kibana_analytics] UPDATE_TARGET_MAPPINGS_PROPERTIES_WAIT_FOR_TASK -> UPDATE_TARGET_MAPPINGS_META. took: 26152ms.
2023-11-11 02:25:21 [2023-11-10T18:25:20.747+00:00][INFO ][savedobjects-service] [.kibana_ingest] UPDATE_TARGET_MAPPINGS_PROPERTIES_WAIT_FOR_TASK -> UPDATE_TARGET_MAPPINGS_META. took: 35102ms.
2023-11-11 02:25:21 [2023-11-10T18:25:21.650+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS_PROPERTIES_WAIT_FOR_TASK -> UPDATE_TARGET_MAPPINGS_META. took: 38028ms.
2023-11-11 02:25:21 [2023-11-10T18:25:21.871+00:00][INFO ][savedobjects-service] [.kibana_security_solution] UPDATE_TARGET_MAPPINGS_PROPERTIES_WAIT_FOR_TASK -> UPDATE_TARGET_MAPPINGS_META. took: 37824ms.
2023-11-11 02:25:34 [2023-11-10T18:25:33.812+00:00][INFO ][savedobjects-service] [.kibana_analytics] UPDATE_TARGET_MAPPINGS_META -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 24302ms.
2023-11-11 02:25:34 [2023-11-10T18:25:34.052+00:00][INFO ][savedobjects-service] [.kibana_analytics] CHECK_VERSION_INDEX_READY_ACTIONS -> MARK_VERSION_INDEX_READY. took: 283ms.
2023-11-11 02:25:36 [2023-11-10T18:25:36.120+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS_META -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 14410ms.
2023-11-11 02:25:36 [2023-11-10T18:25:36.173+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_VERSION_INDEX_READY_ACTIONS -> MARK_VERSION_INDEX_READY. took: 114ms.
2023-11-11 02:25:49 [2023-11-10T18:25:49.182+00:00][INFO ][savedobjects-service] [.kibana_security_solution] UPDATE_TARGET_MAPPINGS_META -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 27258ms.
2023-11-11 02:25:49 [2023-11-10T18:25:49.478+00:00][INFO ][savedobjects-service] [.kibana_security_solution] CHECK_VERSION_INDEX_READY_ACTIONS -> MARK_VERSION_INDEX_READY. took: 351ms.
2023-11-11 02:25:49 [2023-11-10T18:25:49.981+00:00][INFO ][savedobjects-service] [.kibana_ingest] UPDATE_TARGET_MAPPINGS_META -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 29283ms.
2023-11-11 02:25:50 [2023-11-10T18:25:50.056+00:00][INFO ][savedobjects-service] [.kibana_ingest] CHECK_VERSION_INDEX_READY_ACTIONS -> MARK_VERSION_INDEX_READY. took: 75ms.
2023-11-11 02:26:20 [2023-11-10T18:26:20.236+00:00][INFO ][savedobjects-service] [.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 44059ms.
2023-11-11 02:26:20 [2023-11-10T18:26:20.324+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 719223ms
2023-11-11 02:26:20 [2023-11-10T18:26:20.422+00:00][INFO ][savedobjects-service] [.kibana_security_solution] MARK_VERSION_INDEX_READY -> DONE. took: 30944ms.
2023-11-11 02:26:20 [2023-11-10T18:26:20.425+00:00][INFO ][savedobjects-service] [.kibana_security_solution] Migration completed after 718310ms
2023-11-11 02:26:20 [2023-11-10T18:26:20.515+00:00][INFO ][savedobjects-service] [.kibana_ingest] MARK_VERSION_INDEX_READY -> DONE. took: 30460ms.
2023-11-11 02:26:20 [2023-11-10T18:26:20.519+00:00][INFO ][savedobjects-service] [.kibana_ingest] Migration completed after 717501ms
2023-11-11 02:26:20 [2023-11-10T18:26:20.652+00:00][INFO ][savedobjects-service] [.kibana_analytics] MARK_VERSION_INDEX_READY -> DONE. took: 46600ms.
2023-11-11 02:26:20 [2023-11-10T18:26:20.666+00:00][INFO ][savedobjects-service] [.kibana_analytics] Migration completed after 719497ms
2023-11-11 02:26:36 [2023-11-10T18:26:35.871+00:00][INFO ][status] Kibana is now unavailable
2023-11-11 02:26:36 [2023-11-10T18:26:36.052+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
2023-11-11 02:26:36 [2023-11-10T18:26:36.224+00:00][INFO ][plugins-system.preboot] All plugins stopped.
2023-11-11 02:26:36 [2023-11-10T18:26:36.277+00:00][INFO ][plugins-system.standard] Starting [147] plugins: [devTools,translations,share,screenshotMode,usageCollection,telemetryCollectionManager,telemetryCollectionXpack,taskManager,kibanaUsageCollection,cloud,newsfeed,savedObjectsFinder,noDataPage,monitoringCollection,metricsDataAccess,licensing,mapsEms,globalSearch,globalSearchProviders,features,guidedOnboarding,banners,licenseApiGuard,customBranding,ftrApis,fieldFormats,expressions,screenshotting,esUiShared,customIntegrations,contentManagement,dataViews,home,searchprofiler,painlessLab,management,spaces,security,telemetry,licenseManagement,snapshotRestore,lists,files,encryptedSavedObjects,eventLog,actions,notifications,cloudDataMigration,advancedSettings,grokdebugger,console,bfetch,data,savedObjectsTagging,savedObjectsManagement,unifiedSearch,graph,alerting,embeddable,uiActionsEnhanced,savedSearch,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,fileUpload,ingestPipelines,ecsDataQualityDashboard,dataViewFieldEditor,dataViewManagement,charts,watcher,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,visTypeHeatmap,inputControlVis,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,eventAnnotation,expressionXY,dashboard,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,links,lens,maps,discover,reporting,canvas,fleet,logExplorer,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,cases,timelines,sessionView,kubernetesSecurity,cloudDefend,threatIntelligence,osquery,observabilityLogExplorer,observabilityAIAssistant,logsShared,aiops,observability,observabilityOnboarding,ml,uptime,synthetics,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,elasticAssistant,securitySolution,securitySolutionEss,dashboardEnhanced,apmDataAccess,assetManager,apm]
2023-11-11 02:29:43 [2023-11-10T18:29:43.131+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Sender-1.1.3 scheduled with interval 1h
2023-11-11 02:29:43 [2023-11-10T18:29:43.250+00:00][INFO ][plugins.fleet.fleet:check-deleted-files-task:1.0.1] Started with interval of [1d] and timeout of [2m]
2023-11-11 02:29:47 [2023-11-10T18:29:47.453+00:00][INFO ][plugins.monitoring.monitoring] config sourced from: production cluster
2023-11-11 02:29:50 [2023-11-10T18:29:50.131+00:00][INFO ][plugins.observability] Installing SLO shared resources
2023-11-11 02:29:50 [2023-11-10T18:29:50.155+00:00][INFO ][plugins.observability] Installing SLO component template [.slo-observability.sli-mappings]
2023-11-11 02:29:50 [2023-11-10T18:29:50.172+00:00][INFO ][plugins.observability] Installing SLO component template [.slo-observability.sli-settings]
2023-11-11 02:29:50 [2023-11-10T18:29:50.185+00:00][INFO ][plugins.observability] Installing SLO component template [.slo-observability.summary-mappings]
2023-11-11 02:29:50 [2023-11-10T18:29:50.200+00:00][INFO ][plugins.observability] Installing SLO component template [.slo-observability.summary-settings]
2023-11-11 02:29:50 [2023-11-10T18:29:50.660+00:00][INFO ][plugins.alerting] Installing ILM policy .alerts-ilm-policy
2023-11-11 02:29:50 [2023-11-10T18:29:50.781+00:00][INFO ][plugins.alerting] Installing component template .alerts-framework-mappings
2023-11-11 02:29:50 [2023-11-10T18:29:50.832+00:00][INFO ][plugins.alerting] Installing component template .alerts-legacy-alert-mappings
2023-11-11 02:29:51 [2023-11-10T18:29:51.647+00:00][INFO ][plugins.alerting] Installing component template .alerts-ecs-mappings
2023-11-11 02:29:52 [2023-11-10T18:29:52.083+00:00][INFO ][plugins.ruleRegistry] Installing component template .alerts-technical-mappings
2023-11-11 02:29:57 [2023-11-10T18:29:57.559+00:00][INFO ][status] Kibana is now degraded (was unavailable)
2023-11-11 02:34:52 [2023-11-10T18:34:52.561+00:00][INFO ][http.server.Kibana] http server running at http://0.0.0.0:5601
2023-11-11 02:34:57 [2023-11-10T18:34:57.479+00:00][ERROR][plugins.ruleRegistry] Error: Timeout: it took more than 1200000ms
2023-11-11 02:34:57 at Timeout._onTimeout (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:36:18)
2023-11-11 02:34:57 at listOnTimeout (node:internal/timers:569:17)
2023-11-11 02:34:57 at processTimers (node:internal/timers:512:7)
2023-11-11 02:34:57 [2023-11-10T18:34:57.583+00:00][ERROR][plugins.ruleRegistry] Error: Failure during installation of common resources shared between all indices. Timeout: it took more than 1200000ms
2023-11-11 02:34:57 at installWithTimeout (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:48:11)
2023-11-11 02:34:57 at runNextTicks (node:internal/process/task_queues:60:5)
2023-11-11 02:34:57 at processTimers (node:internal/timers:509:9)
2023-11-11 02:34:57 at ResourceInstaller.installCommonResources (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/rule_data_plugin_service/resource_installer.js:42:5)
2023-11-11 02:35:01 [2023-11-10T18:35:01.423+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
2023-11-11 02:35:15 [2023-11-10T18:35:15.229+00:00][INFO ][plugins.telemetry] Telemetry collection is enabled. For more information on telemetry settings, refer to Telemetry settings in Kibana | Kibana Guide [8.11] | Elastic.
2023-11-11 02:35:29 [2023-11-10T18:35:27.509+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Starting monitoring stats collection
2023-11-11 02:35:29 [2023-11-10T18:35:28.881+00:00][INFO ][plugins.fleet] Beginning fleet setup
2023-11-11 02:35:37 [2023-11-10T18:35:36.757+00:00][ERROR][plugins.alerting.usage] Error scheduling Alerts-alerts_invalidate_api_keys task, received socket hang up - Local: 172.18.0.6:36470, Remote: unknown:unknown
2023-11-11 02:35:37 [2023-11-10T18:35:37.192+00:00][ERROR][plugins.osquery.telemetry_events] [task osquery:telemetry-configs:1.1.0]: error scheduling task, received socket hang up - Local: 172.18.0.6:36486, Remote: unknown:unknown
2023-11-11 02:35:37 [2023-11-10T18:35:37.527+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-filterlist-artifact:1.0.0]: error scheduling task, received socket hang up - Local: 172.18.0.6:36502, Remote: unknown:unknown
2023-11-11 02:35:44 [2023-11-10T18:35:44.368+00:00][WARN ][plugins.kibanaUsageCollection] Average event loop delay threshold exceeded 350ms. Received 1853.3594453333333ms. See Task Manager | Kibana Guide [master] | Elastic for more information about scaling Kibana.
2023-11-11 02:35:48 [2023-11-10T18:35:46.256+00:00][ERROR][plugins.apm] Failed to make APM data view available globally
2023-11-11 02:36:09 ConnectionError: socket hang up - Local: 172.18.0.6:36534, Remote: unknown:unknown
2023-11-11 02:36:09 at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:528:31)
2023-11-11 02:36:09 at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-11-11 02:36:09 at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
2023-11-11 02:36:09 at ClientTraced.GetApi [as get] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/get.js:36:12) {
2023-11-11 02:36:09 meta: {
2023-11-11 02:36:09 body: undefined,
2023-11-11 02:36:09 statusCode: 0,
2023-11-11 02:36:09 headers: {},
2023-11-11 02:36:09 meta: {
2023-11-11 02:36:09 context: null,
2023-11-11 02:36:09 request: [Object],
2023-11-11 02:36:09 name: 'elasticsearch-js',
2023-11-11 02:36:09 connection: [Object],
2023-11-11 02:36:09 attempts: 0,
2023-11-11 02:36:09 aborted: false
2023-11-11 02:36:09 },
2023-11-11 02:36:09 warnings: [Getter]
2023-11-11 02:36:09 },
2023-11-11 02:36:09 isBoom: true,
2023-11-11 02:36:09 isServer: true,
2023-11-11 02:36:09 data: null,
2023-11-11 02:36:09 output: {
2023-11-11 02:36:09 statusCode: 503,
2023-11-11 02:36:09 payload: {
2023-11-11 02:36:09 statusCode: 503,
2023-11-11 02:36:09 error: 'Service Unavailable',
2023-11-11 02:36:09 message: 'socket hang up - Local: 172.18.0.6:36534, Remote: unknown:unknown'
2023-11-11 02:36:09 },
2023-11-11 02:36:09 headers: {}
2023-11-11 02:36:09 },
2023-11-11 02:36:09 [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/esUnavailable'
2023-11-11 02:36:09 }
2023-11-11 02:35:50 [2023-11-10T18:35:46.378+00:00][ERROR][plugins.alerting] Error scheduling Alerting-alerting_health_check, received Request timed out
2023-11-11 02:36:09 [2023-11-10T18:35:47.063+00:00][ERROR][plugins.fleet] Error scheduling task, received error: TimeoutError: Request timed out
2023-11-11 02:36:09 [2023-11-10T18:35:47.124+00:00][ERROR][plugins.fleet.fleet:check-deleted-files-task:1.0.1] Error scheduling task, received error: Request timed out
2023-11-11 02:36:09 [2023-11-10T18:35:47.244+00:00][ERROR][plugins.osquery.telemetry_events] [task osquery:telemetry-packs:1.1.0]: error scheduling task, received Request timed out
2023-11-11 02:36:09 [2023-11-10T18:35:47.268+00:00][ERROR][plugins.osquery.telemetry_events] [task osquery:telemetry-saved-queries:1.1.0]: error scheduling task, received Request timed out
2023-11-11 02:36:09 [2023-11-10T18:35:47.353+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:endpoint-diagnostics:1.0.0]: error scheduling task, received Request timed out
2023-11-11 02:36:09 [2023-11-10T18:35:47.403+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:endpoint-meta-telemetry:1.0.0]: error scheduling task, received Request timed out
2023-11-11 02:36:09 [2023-11-10T18:35:47.440+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-lists:1.0.0]: error scheduling task, received Request timed out
2023-11-11 02:36:09 [2023-11-10T18:35:47.559+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-detection-rules:1.0.0]: error scheduling task, received Request timed out
2023-11-11 02:36:09 [2023-11-10T18:35:47.597+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-prebuilt-rule-alerts:1.0.0]: error scheduling task, received Request timed out
2023-11-11 02:36:09 [2023-11-10T18:35:47.624+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-timelines:1.0.0]: error scheduling task, received Request timed out
2023-11-11 02:36:09 [2023-11-10T18:35:47.645+00:00][ERROR][plugins.securitySolution.telemetry_events] [task security:telemetry-configuration:1.0.0]: error scheduling task, received Request timed out
2023-11-11 02:36:09 [2023-11-10T18:35:47.731+00:00][ERROR][plugins.reporting] Error in Reporting start, reporting may not function properly
2023-11-11 02:36:09 [2023-11-10T18:35:47.735+00:00][ERROR][plugins.reporting] TimeoutError: Request timed out
2023-11-11 02:36:09 at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
2023-11-11 02:36:09 at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-11-11 02:36:09 at runNextTicks (node:internal/process/task_queues:64:3)
2023-11-11 02:36:09 at listOnTimeout (node:internal/timers:538:9)
2023-11-11 02:36:09 at processTimers (node:internal/timers:512:7)
2023-11-11 02:36:09 at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
2023-11-11 02:36:09 at ClientTraced.CreateApi [as create] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/create.js:43:12)
2023-11-11 02:36:09 [2023-11-10T18:35:48.148+00:00][ERROR][plugins.apm] Failed to schedule APM source map migration
2023-11-11 02:36:09 [2023-11-10T18:35:48.149+00:00][ERROR][plugins.apm] TimeoutError: Request timed out
2023-11-11 02:36:09 at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
2023-11-11 02:36:09 at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-11-11 02:36:09 at runNextTicks (node:internal/process/task_queues:64:3)
2023-11-11 02:36:09 at listOnTimeout (node:internal/timers:538:9)
2023-11-11 02:36:09 at processTimers (node:internal/timers:512:7)
2023-11-11 02:36:09 at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
2023-11-11 02:36:09 at ClientTraced.CreateApi [as create] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/create.js:43:12)
2023-11-11 02:36:09 [2023-11-10T18:35:49.170+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Logger-Task scheduled with interval 15m
2023-11-11 02:36:09 [2023-11-10T18:35:55.996+00:00][INFO ][plugins.fleet] Agent policies updated by license change: [

2

Pls help me

3

Problem is narrowed down to

image
image1281×174 67.6 KB

image
image997×49 21.7 KB

Unexpected EOF

  1. Kibana container is unhealthy.
  2. It cannot talk to Elasticsearch at Port 9200
  3. Even when I curl request to Port 9200, connection is refused
    image
    image820×90 45.4 KB

I really dont know what to do. This is the 2nd approach i try to setup ELK in Docker and yet face the exact same issue of Kib not talking to ES.

I have even implemented both Inbound and Outbound rules on Ports 9200 and 9201 to allow Traffic in my machine. Yet no improvement.

4

Full Start Up Log

Full Start Up Log

Blockquote
2023-11-11 16:14:53 Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disable see Use Kibana in a production environment | Kibana Guide [8.11] | Elastic
2023-11-11 16:15:01 {"log.level":"info","@timestamp":"2023-11-11T08:15:01.028Z","log":{"logger":"elastic-apm-node"},"agentVersion":"4.0.0","env":{"pid":7,"proctitle":"/usr/share/kibana/bin/../node/bin/node","os":"linux 5.10.102.1-microsoft-standard-WSL2","arch":"x64","host":"b3dfc6147149","timezone":"UTC+00","runtime":"Node.js v18.18.2"},"config":{"serviceName":{"source":"start","value":"kibana","commonName":"service_name"},"serviceVersion":{"source":"start","value":"8.11.0","commonName":"service_version"},"serverUrl":{"source":"start","value":"https://kibana-cloud-apm.apm.us-east-1.aws.found.io/","commonName":"server_url"},"logLevel":{"source":"default","value":"info","commonName":"log_level"},"active":{"source":"start","value":true},"contextPropagationOnly":{"source":"start","value":true},"environment":{"source":"start","value":"production"},"globalLabels":{"source":"start","value":[["kibana_uuid","f02daffe-cc33-4c25-9bcb-321ce7aa0b1d"],["git_rev","f2ea0c43ec0d854259d63d926b97e5c556b5f6b2"]],"sourceValue":{"kibana_uuid":"f02daffe-cc33-4c25-9bcb-321ce7aa0b1d","git_rev":"f2ea0c43ec0d854259d63d926b97e5c556b5f6b2"}},"secretToken":{"source":"start","value":"[REDACTED]","commonName":"secret_token"},"breakdownMetrics":{"source":"start","value":false},"captureSpanStackTraces":{"source":"start","sourceValue":false},"centralConfig":{"source":"start","value":false},"metricsInterval":{"source":"start","value":120,"sourceValue":"120s"},"propagateTracestate":{"source":"start","value":true},"transactionSampleRate":{"source":"start","value":0.1,"commonName":"transaction_sample_rate"},"captureBody":{"source":"start","value":"off","commonName":"capture_body"},"captureHeaders":{"source":"start","value":false}},"activationMethod":"require","ecs":{"version":"1.6.0"},"message":"Elastic APM Node.js Agent v4.0.0"}
2023-11-11 16:15:27 [2023-11-11T08:15:24.386+00:00][INFO ][root] Kibana is starting
2023-11-11 16:15:28 [2023-11-11T08:15:28.459+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
2023-11-11 16:27:59 Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disable see Use Kibana in a production environment | Kibana Guide [8.11] | Elastic
2023-11-11 16:28:10 {"log.level":"info","@timestamp":"2023-11-11T08:28:10.132Z","log":{"logger":"elastic-apm-node"},"agentVersion":"4.0.0","env":{"pid":7,"proctitle":"/usr/share/kibana/bin/../node/bin/node","os":"linux 5.10.102.1-microsoft-standard-WSL2","arch":"x64","host":"b3dfc6147149","timezone":"UTC+00","runtime":"Node.js v18.18.2"},"config":{"serviceName":{"source":"start","value":"kibana","commonName":"service_name"},"serviceVersion":{"source":"start","value":"8.11.0","commonName":"service_version"},"serverUrl":{"source":"start","value":"https://kibana-cloud-apm.apm.us-east-1.aws.found.io/","commonName":"server_url"},"logLevel":{"source":"default","value":"info","commonName":"log_level"},"active":{"source":"start","value":true},"contextPropagationOnly":{"source":"start","value":true},"environment":{"source":"start","value":"production"},"globalLabels":{"source":"start","value":[["kibana_uuid","f02daffe-cc33-4c25-9bcb-321ce7aa0b1d"],["git_rev","f2ea0c43ec0d854259d63d926b97e5c556b5f6b2"]],"sourceValue":{"kibana_uuid":"f02daffe-cc33-4c25-9bcb-321ce7aa0b1d","git_rev":"f2ea0c43ec0d854259d63d926b97e5c556b5f6b2"}},"secretToken":{"source":"start","value":"[REDACTED]","commonName":"secret_token"},"breakdownMetrics":{"source":"start","value":false},"captureSpanStackTraces":{"source":"start","sourceValue":false},"centralConfig":{"source":"start","value":false},"metricsInterval":{"source":"start","value":120,"sourceValue":"120s"},"propagateTracestate":{"source":"start","value":true},"transactionSampleRate":{"source":"start","value":0.1,"commonName":"transaction_sample_rate"},"captureBody":{"source":"start","value":"off","commonName":"capture_body"},"captureHeaders":{"source":"start","value":false}},"activationMethod":"require","ecs":{"version":"1.6.0"},"message":"Elastic APM Node.js Agent v4.0.0"}
2023-11-11 16:29:02 [2023-11-11T08:28:56.353+00:00][INFO ][root] Kibana is starting
2023-11-11 16:29:05 [2023-11-11T08:29:05.871+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
2023-11-11 16:32:20 [2023-11-11T08:32:19.855+00:00][INFO ][plugins-service] Plugin "cloudChat" is disabled.
2023-11-11 16:32:20 [2023-11-11T08:32:20.117+00:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled.
2023-11-11 16:32:20 [2023-11-11T08:32:20.128+00:00][INFO ][plugins-service] Plugin "cloudFullStory" is disabled.
2023-11-11 16:32:20 [2023-11-11T08:32:20.131+00:00][INFO ][plugins-service] Plugin "cloudGainsight" is disabled.
2023-11-11 16:32:20 [2023-11-11T08:32:20.919+00:00][INFO ][plugins-service] Plugin "profilingDataAccess" is disabled.
2023-11-11 16:32:20 [2023-11-11T08:32:20.921+00:00][INFO ][plugins-service] Plugin "profiling" is disabled.
2023-11-11 16:32:21 [2023-11-11T08:32:21.126+00:00][INFO ][plugins-service] Plugin "securitySolutionServerless" is disabled.
2023-11-11 16:32:21 [2023-11-11T08:32:21.126+00:00][INFO ][plugins-service] Plugin "serverless" is disabled.
2023-11-11 16:32:21 [2023-11-11T08:32:21.128+00:00][INFO ][plugins-service] Plugin "serverlessObservability" is disabled.
2023-11-11 16:32:21 [2023-11-11T08:32:21.131+00:00][INFO ][plugins-service] Plugin "serverlessSearch" is disabled.
2023-11-11 16:32:23 [2023-11-11T08:32:23.780+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
2023-11-11 16:32:25 [2023-11-11T08:32:25.648+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
2023-11-11 16:32:26 [2023-11-11T08:32:26.450+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
2023-11-11 16:32:34 [2023-11-11T08:32:34.039+00:00][INFO ][plugins-system.standard] Setting up [147] plugins: [devTools,translations,share,screenshotMode,usageCollection,telemetryCollectionManager,telemetryCollectionXpack,taskManager,kibanaUsageCollection,cloud,newsfeed,savedObjectsFinder,noDataPage,monitoringCollection,metricsDataAccess,licensing,mapsEms,globalSearch,globalSearchProviders,features,guidedOnboarding,banners,licenseApiGuard,customBranding,ftrApis,fieldFormats,expressions,screenshotting,esUiShared,customIntegrations,contentManagement,dataViews,home,searchprofiler,painlessLab,management,spaces,security,telemetry,licenseManagement,snapshotRestore,lists,files,encryptedSavedObjects,eventLog,actions,notifications,cloudDataMigration,advancedSettings,grokdebugger,console,bfetch,data,savedObjectsTagging,savedObjectsManagement,unifiedSearch,graph,alerting,embeddable,uiActionsEnhanced,savedSearch,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,fileUpload,ingestPipelines,ecsDataQualityDashboard,dataViewFieldEditor,dataViewManagement,charts,watcher,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,visTypeHeatmap,inputControlVis,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,eventAnnotation,expressionXY,dashboard,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,links,lens,maps,discover,reporting,canvas,fleet,logExplorer,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,cases,timelines,sessionView,kubernetesSecurity,cloudDefend,threatIntelligence,osquery,observabilityLogExplorer,observabilityAIAssistant,logsShared,aiops,observability,observabilityOnboarding,ml,uptime,synthetics,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,elasticAssistant,securitySolution,securitySolutionEss,dashboardEnhanced,apmDataAccess,assetManager,apm]
2023-11-11 16:32:34 [2023-11-11T08:32:34.830+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: f02daffe-cc33-4c25-9bcb-321ce7aa0b1d
2023-11-11 16:32:36 [2023-11-11T08:32:36.166+00:00][INFO ][custom-branding-service] CustomBrandingService registering plugin: customBranding
2023-11-11 16:32:37 [2023-11-11T08:32:37.529+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
2023-11-11 16:32:37 [2023-11-11T08:32:37.545+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
2023-11-11 16:32:38 [2023-11-11T08:32:38.021+00:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
2023-11-11 16:32:38 [2023-11-11T08:32:38.021+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
2023-11-11 16:32:38 [2023-11-11T08:32:38.649+00:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
2023-11-11 16:32:38 [2023-11-11T08:32:38.775+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
2023-11-11 16:32:38 [2023-11-11T08:32:38.915+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
2023-11-11 16:32:40 [2023-11-11T08:32:40.146+00:00][INFO ][plugins.alerting] using indexes and aliases for persisting alerts
2023-11-11 16:32:40 [2023-11-11T08:32:40.169+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
2023-11-11 16:32:42 [2023-11-11T08:32:42.606+00:00][INFO ][plugins.alerting] Registering resources for context "stack".
2023-11-11 16:32:44 [2023-11-11T08:32:43.983+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
2023-11-11 16:32:44 [2023-11-11T08:32:44.063+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
2023-11-11 16:32:45 [2023-11-11T08:32:45.822+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
2023-11-11 16:32:52 [2023-11-11T08:32:52.286+00:00][INFO ][plugins.alerting] Registering resources for context "observability.slo".
2023-11-11 16:32:52 [2023-11-11T08:32:52.994+00:00][INFO ][plugins.alerting] Registering resources for context "observability.threshold".
2023-11-11 16:32:54 [2023-11-11T08:32:54.059+00:00][INFO ][plugins.alerting] Registering resources for context "ml.anomaly-detection".
2023-11-11 16:32:54 [2023-11-11T08:32:54.272+00:00][INFO ][plugins.alerting] Registering resources for context "observability.uptime".
2023-11-11 16:32:54 [2023-11-11T08:32:54.811+00:00][INFO ][plugins.alerting] Registering resources for context "observability.logs".
2023-11-11 16:32:54 [2023-11-11T08:32:54.818+00:00][INFO ][plugins.alerting] Registering resources for context "observability.metrics".
2023-11-11 16:32:56 [2023-11-11T08:32:56.704+00:00][INFO ][plugins.alerting] Registering resources for context "security".
2023-11-11 16:32:58 [2023-11-11T08:32:58.021+00:00][INFO ][plugins.assetManager] Server is NOT enabled
2023-11-11 16:32:58 [2023-11-11T08:32:58.286+00:00][INFO ][plugins.alerting] Registering resources for context "observability.apm".
2023-11-11 16:32:59 [2023-11-11T08:32:59.168+00:00][WARN ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, but is not supported for Linux Ubuntu 20.04 OS. Automatically setting 'xpack.screenshotting.browser.chromium.disableSandbox: true'.
2023-11-11 16:34:15 [2023-11-11T08:34:15.085+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
2023-11-11 16:34:15 [2023-11-11T08:34:15.196+00:00][INFO ][savedobjects-service] Starting saved objects migrations
2023-11-11 16:34:26 [2023-11-11T08:34:26.839+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 6689ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.096+00:00][INFO ][savedobjects-service] [.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 7142ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.169+00:00][INFO ][savedobjects-service] [.kibana_analytics] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 6675ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.296+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 5436ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.420+00:00][INFO ][savedobjects-service] [.kibana_security_solution] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 5719ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.516+00:00][INFO ][savedobjects-service] [.kibana_ingest] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 5573ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.720+00:00][INFO ][savedobjects-service] [.kibana_security_solution] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 300ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.758+00:00][INFO ][savedobjects-service] [.kibana_security_solution] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 39ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.788+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 492ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.799+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 11ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.817+00:00][INFO ][savedobjects-service] [.kibana] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 722ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.845+00:00][INFO ][savedobjects-service] [.kibana] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 28ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.909+00:00][INFO ][savedobjects-service] [.kibana_ingest] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 394ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.911+00:00][INFO ][savedobjects-service] [.kibana_ingest] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 2ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.931+00:00][INFO ][savedobjects-service] [.kibana_analytics] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 761ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.952+00:00][INFO ][savedobjects-service] [.kibana_analytics] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 21ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.962+00:00][INFO ][savedobjects-service] [.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 1134ms.
2023-11-11 16:34:27 [2023-11-11T08:34:27.970+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 8ms.
2023-11-11 16:34:29 [2023-11-11T08:34:29.497+00:00][INFO ][savedobjects-service] [.kibana_security_solution] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 1736ms.
2023-11-11 16:34:29 [2023-11-11T08:34:29.737+00:00][INFO ][savedobjects-service] [.kibana_analytics] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 1786ms.
2023-11-11 16:34:29 [2023-11-11T08:34:29.812+00:00][INFO ][savedobjects-service] [.kibana_ingest] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 1900ms.
2023-11-11 16:34:31 [2023-11-11T08:34:31.473+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 3502ms.
2023-11-11 16:34:31 [2023-11-11T08:34:31.524+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 3724ms.
2023-11-11 16:34:31 [2023-11-11T08:34:31.557+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 3712ms.
2023-11-11 16:34:31 [2023-11-11T08:34:31.837+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell
2023-11-11 16:34:35 [2023-11-11T08:34:35.348+00:00][INFO ][savedobjects-service] [.kibana_analytics] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 5609ms.
2023-11-11 16:34:35 [2023-11-11T08:34:35.545+00:00][INFO ][savedobjects-service] [.kibana_security_solution] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 6051ms.
2023-11-11 16:34:36 [2023-11-11T08:34:36.419+00:00][INFO ][savedobjects-service] [.kibana_analytics] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 1071ms.
2023-11-11 16:34:36 [2023-11-11T08:34:36.524+00:00][INFO ][savedobjects-service] [.kibana_analytics] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 107ms.
2023-11-11 16:34:36 [2023-11-11T08:34:36.542+00:00][INFO ][savedobjects-service] [.kibana_analytics] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 18ms.
2023-11-11 16:34:36 [2023-11-11T08:34:36.547+00:00][INFO ][savedobjects-service] [.kibana_analytics] Migration completed after 16052ms
2023-11-11 16:34:36 [2023-11-11T08:34:36.641+00:00][INFO ][savedobjects-service] [.kibana_security_solution] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 1096ms.
2023-11-11 16:34:36 [2023-11-11T08:34:36.678+00:00][INFO ][savedobjects-service] [.kibana_security_solution] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 36ms.
2023-11-11 16:34:36 [2023-11-11T08:34:36.702+00:00][INFO ][savedobjects-service] [.kibana_security_solution] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 25ms.
2023-11-11 16:34:36 [2023-11-11T08:34:36.706+00:00][INFO ][savedobjects-service] [.kibana_security_solution] Migration completed after 15005ms
2023-11-11 16:34:39 [2023-11-11T08:34:39.344+00:00][INFO ][savedobjects-service] [.kibana_ingest] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 9530ms.
2023-11-11 16:34:39 [2023-11-11T08:34:39.746+00:00][INFO ][savedobjects-service] [.kibana_ingest] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 404ms.
2023-11-11 16:34:39 [2023-11-11T08:34:39.778+00:00][INFO ][savedobjects-service] [.kibana_ingest] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 33ms.
2023-11-11 16:34:39 [2023-11-11T08:34:39.794+00:00][INFO ][savedobjects-service] [.kibana_ingest] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 16ms.
2023-11-11 16:34:39 [2023-11-11T08:34:39.798+00:00][INFO ][savedobjects-service] [.kibana_ingest] Migration completed after 17855ms
2023-11-11 16:34:40 [2023-11-11T08:34:40.422+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 8899ms.
2023-11-11 16:34:40 [2023-11-11T08:34:40.508+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 9035ms.
2023-11-11 16:34:40 [2023-11-11T08:34:40.572+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 9014ms.
2023-11-11 16:34:40 [2023-11-11T08:34:40.617+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 109ms.
2023-11-11 16:34:40 [2023-11-11T08:34:40.639+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 22ms.
2023-11-11 16:34:40 [2023-11-11T08:34:40.653+00:00][INFO ][savedobjects-service] [.kibana_task_manager] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 15ms.
2023-11-11 16:34:40 [2023-11-11T08:34:40.658+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 20516ms
2023-11-11 16:34:41 [2023-11-11T08:34:41.639+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 1216ms.
2023-11-11 16:34:41 [2023-11-11T08:34:41.660+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 22ms.
2023-11-11 16:34:41 [2023-11-11T08:34:41.674+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 14ms.
2023-11-11 16:34:41 [2023-11-11T08:34:41.676+00:00][INFO ][savedobjects-service] [.kibana_alerting_cases] Migration completed after 19816ms
2023-11-11 16:34:41 [2023-11-11T08:34:41.728+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> CHECK_TARGET_MAPPINGS. took: 1156ms.
2023-11-11 16:34:41 [2023-11-11T08:34:41.750+00:00][INFO ][savedobjects-service] [.kibana] CHECK_TARGET_MAPPINGS -> CHECK_VERSION_INDEX_READY_ACTIONS. took: 23ms.
2023-11-11 16:34:41 [2023-11-11T08:34:41.766+00:00][INFO ][savedobjects-service] [.kibana] CHECK_VERSION_INDEX_READY_ACTIONS -> DONE. took: 16ms.
2023-11-11 16:34:41 [2023-11-11T08:34:41.767+00:00][INFO ][savedobjects-service] [.kibana] Migration completed after 21814ms
2023-11-11 16:34:42 [2023-11-11T08:34:42.788+00:00][INFO ][status] Kibana is now unavailable
2023-11-11 16:34:42 [2023-11-11T08:34:42.857+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
2023-11-11 16:34:42 [2023-11-11T08:34:42.931+00:00][INFO ][plugins-system.preboot] All plugins stopped.
2023-11-11 16:34:42 [2023-11-11T08:34:42.982+00:00][INFO ][plugins-system.standard] Starting [147] plugins: [devTools,translations,share,screenshotMode,usageCollection,telemetryCollectionManager,telemetryCollectionXpack,taskManager,kibanaUsageCollection,cloud,newsfeed,savedObjectsFinder,noDataPage,monitoringCollection,metricsDataAccess,licensing,mapsEms,globalSearch,globalSearchProviders,features,guidedOnboarding,banners,licenseApiGuard,customBranding,ftrApis,fieldFormats,expressions,screenshotting,esUiShared,customIntegrations,contentManagement,dataViews,home,searchprofiler,painlessLab,management,spaces,security,telemetry,licenseManagement,snapshotRestore,lists,files,encryptedSavedObjects,eventLog,actions,notifications,cloudDataMigration,advancedSettings,grokdebugger,console,bfetch,data,savedObjectsTagging,savedObjectsManagement,unifiedSearch,graph,alerting,embeddable,uiActionsEnhanced,savedSearch,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,fileUpload,ingestPipelines,ecsDataQualityDashboard,dataViewFieldEditor,dataViewManagement,charts,watcher,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,visTypeHeatmap,inputControlVis,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,eventAnnotation,expressionXY,dashboard,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,links,lens,maps,discover,reporting,canvas,fleet,logExplorer,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,dataVisualizer,cases,timelines,sessionView,kubernetesSecurity,cloudDefend,threatIntelligence,osquery,observabilityLogExplorer,observabilityAIAssistant,logsShared,aiops,observability,observabilityOnboarding,ml,uptime,synthetics,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,elasticAssistant,securitySolution,securitySolutionEss,dashboardEnhanced,apmDataAccess,assetManager,apm]
2023-11-11 16:34:50 [2023-11-11T08:34:50.463+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Sender-1.1.3 scheduled with interval 1h
2023-11-11 16:34:50 [2023-11-11T08:34:50.604+00:00][INFO ][plugins.fleet.fleet:check-deleted-files-task:1.0.1] Started with interval of [1d] and timeout of [2m]
2023-11-11 16:34:51 [2023-11-11T08:34:51.634+00:00][INFO ][plugins.monitoring.monitoring] config sourced from: production cluster
2023-11-11 16:34:55 [2023-11-11T08:34:55.210+00:00][INFO ][plugins.observability] Installing SLO shared resources
2023-11-11 16:34:55 [2023-11-11T08:34:55.243+00:00][INFO ][plugins.observability] Installing SLO component template [.slo-observability.sli-mappings]
2023-11-11 16:34:55 [2023-11-11T08:34:55.263+00:00][INFO ][plugins.observability] Installing SLO component template [.slo-observability.sli-settings]
2023-11-11 16:34:55 [2023-11-11T08:34:55.290+00:00][INFO ][plugins.observability] Installing SLO component template [.slo-observability.summary-mappings]
2023-11-11 16:34:55 [2023-11-11T08:34:55.306+00:00][INFO ][plugins.observability] Installing SLO component template [.slo-observability.summary-settings]
2023-11-11 16:34:55 [2023-11-11T08:34:55.547+00:00][INFO ][plugins.alerting] Installing ILM policy .alerts-ilm-policy
2023-11-11 16:34:55 [2023-11-11T08:34:55.623+00:00][INFO ][plugins.alerting] Installing component template .alerts-framework-mappings
2023-11-11 16:34:55 [2023-11-11T08:34:55.656+00:00][INFO ][plugins.alerting] Installing component template .alerts-legacy-alert-mappings
2023-11-11 16:34:56 [2023-11-11T08:34:56.321+00:00][INFO ][plugins.alerting] Installing component template .alerts-ecs-mappings
2023-11-11 16:34:56 [2023-11-11T08:34:56.768+00:00][INFO ][plugins.ruleRegistry] Installing component template .alerts-technical-mappings
unexpected EOF

5

If you cannot talk with Elasticsearch this could means that your cluster is not working correctly, you need to check the Logs for your Elasticsearch nodes.

Kibana won't work if the Elasticsearch cluster is not running, before troubleshoot Kibana you need to check if your cluster is running.

Also, you didn't share your full docker-compose.yml file, please share your full configuration file.

And while the video you shared uses the official documentation, I recommend that you also check the same documentation.

6

Alright.

[details="docker-compose.yml"] FORMATTED

version: "3.8"

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  esdata02:
    driver: local
  esdata03:
    driver: local
  kibanadata:
    driver: local
  logstashdata01:
    driver: local

networks:
  default:
    name: elastic
    external: false
    
services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es02\n"\
          "    dns:\n"\
          "      - es02\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es03\n"\
          "    dns:\n"\
          "      - es03\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: kibana\n"\
          "    dns:\n"\
          "      - kibana\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - 9201:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es02,es03
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${ES_MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es02:
    depends_on:
      - es01
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata02:/usr/share/elasticsearch/data
    environment:
      - node.name=es02
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es02/es02.key
      - xpack.security.http.ssl.certificate=certs/es02/es02.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es02/es02.key
      - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${ES_MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es03:
    depends_on:
      - es02
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata03:/usr/share/elasticsearch/data
    environment:
      - node.name=es03
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es02
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es03/es03.key
      - xpack.security.http.ssl.certificate=certs/es03/es03.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es03/es03.key
      - xpack.security.transport.ssl.certificate=certs/es03/es03.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${ES_MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
      # es02:
      #   condition: service_healthy
      # es03:
      #   condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    labels:
      co.elastic.logs/module: kibana
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
    mem_limit: ${KB_MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 5s
      timeout: 10s
      retries: 10

  logstash:
    depends_on: 
      es01:
        condition: service_healthy
      kibana:
        condition: service_healthy
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    volumes:
      - logstashdata01:/usr/share/logstash/data
      - certs:/usr/share/logstash/certs
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
    environment:
      - NODE_NAME="logstash"
      - xpack.monitoring.enabled=false
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD={ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://es01:9200
    command: logstash -f /usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044/udp"
    mem_limit: ${LS_MEM_LIMIT}

[/details]

Btw, even tho the ubuntu CLI says that all my ES containers are Healthy, it still can mean something is wrong?
image

7

Please format your code for the docker-compose.yml, use the Preformatted text option, the </>, it is pretty hard to read without proper formatting.

Also, do not use the hide option, there is no need.

This just means that the container is running, the service on the container can have some error, did you check the the logs for the elasticsearch containers? You need to check and share the logs.

8

Your issue seems to be a wrong docker configuration.

You are binding the host port 9201 to the container port 9200 for the container es01.

But in your Kibana container you are using the port 9200

This won't work because the por 9200 is not exposed on your host, just the 9201.

Not sure why you changed that, the official docker compose uses this:

    ports:
      - ${ES_PORT}:9200

So the port will be populated by the value on the env file.

You need to fix that, or you use change to use the port from the env file or you change your Kibana configuration to use port 9201.

1 Like
9

Read your post and let me run it again. Thanks for the points out.

ES LOGS

2023-11-11 21:35:48 {"@timestamp":"2023-11-11T13:35:48.311Z", "log.level": "WARN", "message":"master not discovered or elected yet, an election requires at least 2 nodes with ids from [TMQPiSCqQeSMOiDsX_C5MQ, aSKFSYSvR96IAjii9hFj4w, -JiMNlfRSOi4PBvbLV7urg], have discovered possible quorum [{es01}{TMQPiSCqQeSMOiDsX_C5MQ}{P0QZODU9TN6URctBXxcDug}{es01}{172.23.0.3}{172.23.0.3:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}, {es02}{-JiMNlfRSOi4PBvbLV7urg}{oTCucY-4RVqJqK1muV3_5Q}{es02}{172.23.0.4}{172.23.0.4:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}, {es03}{aSKFSYSvR96IAjii9hFj4w}{025EJTvXQlSOUAr5Fn-vZg}{es03}{172.23.0.5}{172.23.0.5:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}]; discovery will continue using [172.23.0.4:9300, 172.23.0.5:9300] from hosts providers and [{es01}{TMQPiSCqQeSMOiDsX_C5MQ}{P0QZODU9TN6URctBXxcDug}{es01}{172.23.0.3}{172.23.0.3:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}] from last-known cluster state; node term 22, last-accepted version 257 in term 19; joining [{es02}{-JiMNlfRSOi4PBvbLV7urg}{oTCucY-4RVqJqK1muV3_5Q}{es02}{172.23.0.4}{172.23.0.4:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}] in term [22] has status [waiting for response] after [200ms]; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.11/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 21:35:54 {"@timestamp":"2023-11-11T13:35:54.514Z", "log.level": "INFO", "message":"master node changed {previous [], current [{es02}{-JiMNlfRSOi4PBvbLV7urg}{oTCucY-4RVqJqK1muV3_5Q}{es02}{172.23.0.4}{172.23.0.4:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}]}, added {{es02}{-JiMNlfRSOi4PBvbLV7urg}{oTCucY-4RVqJqK1muV3_5Q}{es02}{172.23.0.4}{172.23.0.4:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}, {es03}{aSKFSYSvR96IAjii9hFj4w}{025EJTvXQlSOUAr5Fn-vZg}{es03}{172.23.0.5}{172.23.0.5:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}}, term: 22, version: 293, reason: ApplyCommitRequest{term=22, version=293, sourceNode={es02}{-JiMNlfRSOi4PBvbLV7urg}{oTCucY-4RVqJqK1muV3_5Q}{es02}{172.23.0.4}{172.23.0.4:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}{ml.allocated_processors_double=8.0, ml.max_jvm_size=2147483648, ml.config_version=11.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=4294967296, ml.allocated_processors=8}}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.ClusterApplierService","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 21:35:57 {"@timestamp":"2023-11-11T13:35:57.458Z", "log.level": "INFO", "message":"refresh keys", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.TokenService","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 21:35:57 {"@timestamp":"2023-11-11T13:35:57.909Z", "log.level": "INFO", "message":"refreshed keys", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.TokenService","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 21:35:58 {"@timestamp":"2023-11-11T13:35:58.679Z", "log.level": "INFO", "message":"license [4a2d1b11-2c62-4962-834d-5090a8f205dc] mode [basic] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 21:35:58 {"@timestamp":"2023-11-11T13:35:58.705Z", "log.level": "INFO", "message":"license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 21:35:59 {"@timestamp":"2023-11-11T13:35:59.011Z", "log.level": "INFO", "message":"publish_address {172.23.0.3:9200}, bound_addresses {0.0.0.0:9200}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.http.AbstractHttpServerTransport","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 21:35:59 {"@timestamp":"2023-11-11T13:35:59.086Z", "log.level": "INFO", "message":"started {es01}{TMQPiSCqQeSMOiDsX_C5MQ}{P0QZODU9TN6URctBXxcDug}{es01}{172.23.0.3}{172.23.0.3:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}{ml.allocated_processors_double=8.0, ml.max_jvm_size=2147483648, ml.config_version=11.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=4294967296, ml.allocated_processors=8}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 21:36:09 {"@timestamp":"2023-11-11T13:36:08.993Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#4]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","tags":[" [.kibana_security_solution_8.11.0_001]"]}
2023-11-11 21:36:09 {"@timestamp":"2023-11-11T13:36:09.207Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#1]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","tags":[" [.kibana_8.11.0_001]"]}
2023-11-11 21:37:14 {"@timestamp":"2023-11-11T13:37:14.034Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#4]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","tags":[" [.security-7]"]}
2023-11-11 21:37:16 {"@timestamp":"2023-11-11T13:37:16.189Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#4]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","tags":[" [.kibana_task_manager_8.11.0_001]"]}
2023-11-11 21:37:35 {"@timestamp":"2023-11-11T13:37:35.724Z", "log.level": "WARN", "message":"absolute clock went backwards by [441ms/441ms] while timer thread was sleeping", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][[timer]]","log.logger":"org.elasticsearch.threadpool.ThreadPool","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 21:37:41 {"@timestamp":"2023-11-11T13:37:41.245Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#1]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","tags":[" [.tasks]"]}
2023-11-11 21:37:41 {"@timestamp":"2023-11-11T13:37:41.568Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#4]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","tags":[" [.apm-custom-link]"]}
2023-11-11 21:39:08 {"@timestamp":"2023-11-11T13:39:08.185Z", "log.level": "INFO", "message":"[gc][young][213][11] duration [742ms], collections [1]/[1s], total [742ms]/[2.1s], memory [516.4mb]->[116.5mb]/[2gb], all_pools {[young] [436mb]->[4mb]/[0b]}{[old] [64.5mb]->[64.5mb]/[2gb]}{[survivor] [15.9mb]->[48mb]/[0b]}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 21:39:08 {"@timestamp":"2023-11-11T13:39:08.224Z", "log.level": "WARN", "message":"[gc][213] overhead, spent [742ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 21:40:25 {"@timestamp":"2023-11-11T13:40:25.638Z", "log.level": "WARN", "message":"[gc][young][289][12] duration [1.8s], collections [1]/[2s], total [1.8s]/[3.9s], memory [160.5mb]->[114.9mb]/[2gb], all_pools {[young] [48mb]->[0b]/[0b]}{[old] [64.5mb]->[110.2mb]/[2gb]}{[survivor] [48mb]->[4.6mb]/[0b]}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 21:40:25 {"@timestamp":"2023-11-11T13:40:25.712Z", "log.level": "WARN", "message":"[gc][289] overhead, spent [1.8s] collecting in the last [2s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 21:43:30 {"@timestamp":"2023-11-11T13:43:30.193Z", "log.level": "INFO", "message":"[gc][young][470][13] duration [767ms], collections [1]/[1.4s], total [767ms]/[4.7s], memory [338.9mb]->[113.5mb]/[2gb], all_pools {[young] [224mb]->[0b]/[0b]}{[old] [110.2mb]->[110.2mb]/[2gb]}{[survivor] [4.6mb]->[3.2mb]/[0b]}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 21:43:30 {"@timestamp":"2023-11-11T13:43:30.350Z", "log.level": "WARN", "message":"[gc][470] overhead, spent [767ms] collecting in the last [1.4s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
10

Managed to now get es01 failed. Latest logs attached.
LATEST ES-LOGS
image

2023-11-11 22:01:13 {"@timestamp":"2023-11-11T14:01:13.327Z", "log.level": "INFO", "message":"bound or publishing to a non-loopback address, enforcing bootstrap checks", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.BootstrapChecks","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 22:01:14 {"@timestamp":"2023-11-11T14:01:14.511Z", "log.level": "WARN", "message":"this node is locked into cluster UUID [3m-f4WNRREeLwEQNJDIbbQ] but [cluster.initial_master_nodes] is set to [es01, es02, es03]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.11/important-settings.html#initial_master_nodes", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.cluster.coordination.ClusterBootstrapService","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 22:01:26 {"@timestamp":"2023-11-11T14:01:26.688Z", "log.level": "WARN", "message":"master not discovered or elected yet, an election requires at least 2 nodes with ids from [TMQPiSCqQeSMOiDsX_C5MQ, aSKFSYSvR96IAjii9hFj4w, -JiMNlfRSOi4PBvbLV7urg], have only discovered non-quorum [{es01}{TMQPiSCqQeSMOiDsX_C5MQ}{wK6SjdGqQJaXnNZDCcaZlQ}{es01}{172.23.0.3}{172.23.0.3:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}]; discovery will continue using [172.23.0.4:9300, 172.23.0.5:9300] from hosts providers and [{es01}{TMQPiSCqQeSMOiDsX_C5MQ}{wK6SjdGqQJaXnNZDCcaZlQ}{es01}{172.23.0.3}{172.23.0.3:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}] from last-known cluster state; node term 22, last-accepted version 302 in term 22; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.11/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 22:01:37 {"@timestamp":"2023-11-11T14:01:37.282Z", "log.level": "WARN", "message":"master not discovered or elected yet, an election requires at least 2 nodes with ids from [TMQPiSCqQeSMOiDsX_C5MQ, aSKFSYSvR96IAjii9hFj4w, -JiMNlfRSOi4PBvbLV7urg], have only discovered non-quorum [{es01}{TMQPiSCqQeSMOiDsX_C5MQ}{wK6SjdGqQJaXnNZDCcaZlQ}{es01}{172.23.0.3}{172.23.0.3:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}]; discovery will continue using [172.23.0.4:9300, 172.23.0.5:9300] from hosts providers and [{es01}{TMQPiSCqQeSMOiDsX_C5MQ}{wK6SjdGqQJaXnNZDCcaZlQ}{es01}{172.23.0.3}{172.23.0.3:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}] from last-known cluster state; node term 22, last-accepted version 302 in term 22; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.11/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 22:01:46 {"@timestamp":"2023-11-11T14:01:46.092Z", "log.level": "WARN", "message":"timed out while waiting for initial discovery state - timeout: 30s", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#1]","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 22:01:46 {"@timestamp":"2023-11-11T14:01:46.748Z", "log.level": "INFO", "message":"publish_address {172.23.0.3:9200}, bound_addresses {0.0.0.0:9200}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.http.AbstractHttpServerTransport","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 22:01:46 {"@timestamp":"2023-11-11T14:01:46.758Z", "log.level": "INFO", "message":"started {es01}{TMQPiSCqQeSMOiDsX_C5MQ}{wK6SjdGqQJaXnNZDCcaZlQ}{es01}{172.23.0.3}{172.23.0.3:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}{ml.allocated_processors=8, ml.machine_memory=4294967296, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=2147483648, ml.allocated_processors_double=8.0}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 22:01:50 {"@timestamp":"2023-11-11T14:01:50.073Z", "log.level": "WARN", "message":"master not discovered or elected yet, an election requires at least 2 nodes with ids from [TMQPiSCqQeSMOiDsX_C5MQ, aSKFSYSvR96IAjii9hFj4w, -JiMNlfRSOi4PBvbLV7urg], have discovered possible quorum [{es01}{TMQPiSCqQeSMOiDsX_C5MQ}{wK6SjdGqQJaXnNZDCcaZlQ}{es01}{172.23.0.3}{172.23.0.3:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}, {es02}{-JiMNlfRSOi4PBvbLV7urg}{C0zycprTTEWD4b5Qy8cDpA}{es02}{172.23.0.4}{172.23.0.4:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}, {es03}{aSKFSYSvR96IAjii9hFj4w}{iUBcrF_ITWKAsdDadIHSew}{es03}{172.23.0.5}{172.23.0.5:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}]; discovery will continue using [172.23.0.4:9300, 172.23.0.5:9300] from hosts providers and [{es01}{TMQPiSCqQeSMOiDsX_C5MQ}{wK6SjdGqQJaXnNZDCcaZlQ}{es01}{172.23.0.3}{172.23.0.3:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}] from last-known cluster state; node term 24, last-accepted version 302 in term 22; joining [{es02}{-JiMNlfRSOi4PBvbLV7urg}{C0zycprTTEWD4b5Qy8cDpA}{es02}{172.23.0.4}{172.23.0.4:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}] in term [24] has status [waiting for response] after [6s/6096ms]; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.11/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 22:01:56 {"@timestamp":"2023-11-11T14:01:56.350Z", "log.level": "INFO", "message":"master node changed {previous [], current [{es02}{-JiMNlfRSOi4PBvbLV7urg}{C0zycprTTEWD4b5Qy8cDpA}{es02}{172.23.0.4}{172.23.0.4:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}]}, added {{es03}{aSKFSYSvR96IAjii9hFj4w}{iUBcrF_ITWKAsdDadIHSew}{es03}{172.23.0.5}{172.23.0.5:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}, {es02}{-JiMNlfRSOi4PBvbLV7urg}{C0zycprTTEWD4b5Qy8cDpA}{es02}{172.23.0.4}{172.23.0.4:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}}, term: 24, version: 319, reason: ApplyCommitRequest{term=24, version=319, sourceNode={es02}{-JiMNlfRSOi4PBvbLV7urg}{C0zycprTTEWD4b5Qy8cDpA}{es02}{172.23.0.4}{172.23.0.4:9300}{cdfhilmrstw}{8.11.0}{7000099-8500003}{ml.allocated_processors=8, ml.machine_memory=4294967296, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=2147483648, ml.allocated_processors_double=8.0}}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.ClusterApplierService","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 22:01:58 {"@timestamp":"2023-11-11T14:01:58.026Z", "log.level": "INFO", "message":"retrying master election after [10] failed attempts; election attempts are currently scheduled up to [1100ms] apart", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ElectionSchedulerFactory","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 22:02:01 {"@timestamp":"2023-11-11T14:02:01.781Z", "log.level": "INFO", "message":"refresh keys", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.TokenService","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 22:02:02 {"@timestamp":"2023-11-11T14:02:02.640Z", "log.level": "INFO", "message":"refreshed keys", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.TokenService","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 22:02:04 {"@timestamp":"2023-11-11T14:02:04.083Z", "log.level": "INFO", "message":"license [4a2d1b11-2c62-4962-834d-5090a8f205dc] mode [basic] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.ClusterStateLicenseService","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 22:02:04 {"@timestamp":"2023-11-11T14:02:04.119Z", "log.level": "INFO", "message":"license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
2023-11-11 22:02:09 {"@timestamp":"2023-11-11T14:02:09.467Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#1]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","tags":[" [.kibana_security_solution_8.11.0_001]"]}
2023-11-11 22:02:09 {"@timestamp":"2023-11-11T14:02:09.501Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#2]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","tags":[" [.kibana_task_manager_8.11.0_001]"]}
2023-11-11 22:02:13 {"@timestamp":"2023-11-11T14:02:13.595Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#3]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","tags":[" [.security-7]"]}
2023-11-11 22:02:13 {"@timestamp":"2023-11-11T14:02:13.709Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#2]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","tags":[" [.kibana_8.11.0_001]"]}
2023-11-11 22:02:16 {"@timestamp":"2023-11-11T14:02:16.631Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#1]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","tags":[" [.apm-custom-link]"]}
2023-11-11 22:02:18 {"@timestamp":"2023-11-11T14:02:18.366Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#5]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","tags":[" [.tasks]"]}
2023-11-11 22:02:24 {"@timestamp":"2023-11-11T14:02:24.128Z", "log.level": "INFO", "message":"reloading search analyzers", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#3]","log.logger":"org.elasticsearch.index.mapper.MapperService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster","tags":[" [.kibana_analytics_8.11.0_001]"]}
2023-11-11 22:13:33 {"@timestamp":"2023-11-11T14:13:32.999Z", "log.level": "INFO", "message":"[gc][743] overhead, spent [411ms] collecting in the last [1s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][scheduler][T#1]","log.logger":"org.elasticsearch.monitor.jvm.JvmGcMonitorService","elasticsearch.cluster.uuid":"3m-f4WNRREeLwEQNJDIbbQ","elasticsearch.node.id":"TMQPiSCqQeSMOiDsX_C5MQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}

DOCKER-COMPOSE

version: "3.8"

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  esdata02:
    driver: local
  esdata03:
    driver: local
  kibanadata:
    driver: local
  logstashdata01:
    driver: local

networks:
  default:
    name: elastic
    external: false
    
services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es02\n"\
          "    dns:\n"\
          "      - es02\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es03\n"\
          "    dns:\n"\
          "      - es03\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: kibana\n"\
          "    dns:\n"\
          "      - kibana\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - 9201:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es02,es03
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${ES_MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9201 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es02:
    depends_on:
      - es01
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata02:/usr/share/elasticsearch/data
    environment:
      - node.name=es02
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es02/es02.key
      - xpack.security.http.ssl.certificate=certs/es02/es02.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es02/es02.key
      - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${ES_MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9201 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es03:
    depends_on:
      - es02
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata03:/usr/share/elasticsearch/data
    environment:
      - node.name=es03
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es02
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es03/es03.key
      - xpack.security.http.ssl.certificate=certs/es03/es03.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es03/es03.key
      - xpack.security.transport.ssl.certificate=certs/es03/es03.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${ES_MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9201 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
      # es02:
      #   condition: service_healthy
      # es03:
      #   condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    labels:
      co.elastic.logs/module: kibana
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
      - ${ES_PORT}:9200
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9201
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
    mem_limit: ${KB_MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 5s
      timeout: 10s
      retries: 10

  logstash:
    depends_on: 
      es01:
        condition: service_healthy
      kibana:
        condition: service_healthy
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    volumes:
      - logstashdata01:/usr/share/logstash/data
      - certs:/usr/share/logstash/certs
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
    environment:
      - NODE_NAME="logstash"
      - xpack.monitoring.enabled=false
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD={ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://es01:9201
    command: logstash -f /usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044/udp"
    mem_limit: ${LS_MEM_LIMIT}
11

This is wrong, please check the official docker compose and validate with your compose.

This is the configuration for your kibana container, you only need to expose the kibana port, remove the - ${ES_PORT}:9200 line from this part of your compose.

Rollback the changes you made to your docker compose, you need to only change this:

The ports needs to be - ${ES_PORT}:9200

Your issue is a docker issue, you need to validate your docker compose to see if all ports are correctly exposed.

You need to change all 9201 port back to 9200, and use the ES_PORT variable in your es01 container configuration only.

Then you set this port to 9200 in your env file.

12

Looking with more attention, the external port should not make any difference in this case.

But try with this docker compose to see if it works.

version: "3.8"

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  esdata02:
    driver: local
  esdata03:
    driver: local
  kibanadata:
    driver: local
  logstashdata01:
    driver: local

networks:
  default:
    name: elastic
    external: false
    
services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es02\n"\
          "    dns:\n"\
          "      - es02\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es03\n"\
          "    dns:\n"\
          "      - es03\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: kibana\n"\
          "    dns:\n"\
          "      - kibana\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es02,es03
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${ES_MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es02:
    depends_on:
      - es01
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata02:/usr/share/elasticsearch/data
    environment:
      - node.name=es02
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es02/es02.key
      - xpack.security.http.ssl.certificate=certs/es02/es02.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es02/es02.key
      - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${ES_MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es03:
    depends_on:
      - es02
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata03:/usr/share/elasticsearch/data
    environment:
      - node.name=es03
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es02
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es03/es03.key
      - xpack.security.http.ssl.certificate=certs/es03/es03.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es03/es03.key
      - xpack.security.transport.ssl.certificate=certs/es03/es03.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${ES_MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
      # es02:
      #   condition: service_healthy
      # es03:
      #   condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    labels:
      co.elastic.logs/module: kibana
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
    mem_limit: ${KB_MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 5s
      timeout: 10s
      retries: 10

  logstash:
    depends_on: 
      es01:
        condition: service_healthy
      kibana:
        condition: service_healthy
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    volumes:
      - logstashdata01:/usr/share/logstash/data
      - certs:/usr/share/logstash/certs
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
    environment:
      - NODE_NAME="logstash"
      - xpack.monitoring.enabled=false
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD={ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://es01:9200
    command: logstash -f /usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044/udp"
    mem_limit: ${LS_MEM_LIMIT}
1 Like
13

What was the edit u made here?

14

Just the port for the es01 container, but nevermind, I don't think this is the issue as this is just related to the port on the host, and kibana uses the internal port.

I'm trying to run your compose to see if I catch the error.

15

image
In ubuntu CLI, the Kib container threw Error

But in Docker, Kib Container is still running.

image
image1274×425 221 KB
\

2023-11-11 22:57:15 [2023-11-11T14:57:15.128+00:00][WARN ][environment] Detected an unhandled Promise rejection: TimeoutError: Request timed out
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
2023-11-11 22:57:15     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-11-11 22:57:15     at runNextTicks (node:internal/process/task_queues:64:3)
2023-11-11 22:57:15     at listOnTimeout (node:internal/timers:538:9)
2023-11-11 22:57:15     at processTimers (node:internal/timers:512:7)
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
2023-11-11 22:57:15     at ClientTraced.GetApi [as get] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/get.js:36:12)
2023-11-11 22:57:15 [2023-11-11T14:57:15.162+00:00][WARN ][process] UnhandledPromiseRejectionWarning: TimeoutError: Request timed out
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
2023-11-11 22:57:15     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-11-11 22:57:15     at runNextTicks (node:internal/process/task_queues:64:3)
2023-11-11 22:57:15     at listOnTimeout (node:internal/timers:538:9)
2023-11-11 22:57:15     at processTimers (node:internal/timers:512:7)
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
2023-11-11 22:57:15     at ClientTraced.GetApi [as get] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/get.js:36:12)
2023-11-11 22:57:15     at emitUnhandledRejectionWarning (node:internal/process/promises:200:15)
2023-11-11 22:57:15     at processPromiseRejections (node:internal/process/promises:279:11)
2023-11-11 22:57:15     at processTicksAndRejections (node:internal/process/task_queues:96:32)
2023-11-11 22:57:15     at runNextTicks (node:internal/process/task_queues:64:3)
2023-11-11 22:57:15     at listOnTimeout (node:internal/timers:538:9)
2023-11-11 22:57:15     at processTimers (node:internal/timers:512:7)
2023-11-11 22:57:15 [2023-11-11T14:57:15.167+00:00][WARN ][process] TimeoutError: Request timed out
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
2023-11-11 22:57:15     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-11-11 22:57:15     at runNextTicks (node:internal/process/task_queues:64:3)
2023-11-11 22:57:15     at listOnTimeout (node:internal/timers:538:9)
2023-11-11 22:57:15     at processTimers (node:internal/timers:512:7)
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
2023-11-11 22:57:15     at ClientTraced.GetApi [as get] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/get.js:36:12)
2023-11-11 22:57:15 [2023-11-11T14:57:15.184+00:00][ERROR][plugins.reporting] Error in Reporting start, reporting may not function properly
2023-11-11 22:57:15 [2023-11-11T14:57:15.186+00:00][ERROR][plugins.reporting] TimeoutError: Request timed out
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
2023-11-11 22:57:15     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-11-11 22:57:15     at runNextTicks (node:internal/process/task_queues:64:3)
2023-11-11 22:57:15     at listOnTimeout (node:internal/timers:538:9)
2023-11-11 22:57:15     at processTimers (node:internal/timers:512:7)
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
2023-11-11 22:57:15     at ClientTraced.CreateApi [as create] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/create.js:43:12)
2023-11-11 22:57:15 [2023-11-11T14:57:15.201+00:00][WARN ][environment] Detected an unhandled Promise rejection: TimeoutError: Request timed out
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
2023-11-11 22:57:15     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-11-11 22:57:15     at runNextTicks (node:internal/process/task_queues:64:3)
2023-11-11 22:57:15     at listOnTimeout (node:internal/timers:538:9)
2023-11-11 22:57:15     at processTimers (node:internal/timers:512:7)
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
2023-11-11 22:57:15     at ClientTraced.GetApi [as get] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/get.js:36:12)
2023-11-11 22:57:15 [2023-11-11T14:57:15.204+00:00][WARN ][process] UnhandledPromiseRejectionWarning: TimeoutError: Request timed out
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
2023-11-11 22:57:15     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-11-11 22:57:15     at runNextTicks (node:internal/process/task_queues:64:3)
2023-11-11 22:57:15     at listOnTimeout (node:internal/timers:538:9)
2023-11-11 22:57:15     at processTimers (node:internal/timers:512:7)
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
2023-11-11 22:57:15     at ClientTraced.GetApi [as get] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/get.js:36:12)
2023-11-11 22:57:15     at emitUnhandledRejectionWarning (node:internal/process/promises:200:15)
2023-11-11 22:57:15     at processPromiseRejections (node:internal/process/promises:279:11)
2023-11-11 22:57:15     at processTicksAndRejections (node:internal/process/task_queues:96:32)
2023-11-11 22:57:15     at runNextTicks (node:internal/process/task_queues:64:3)
2023-11-11 22:57:15     at listOnTimeout (node:internal/timers:538:9)
2023-11-11 22:57:15     at processTimers (node:internal/timers:512:7)
2023-11-11 22:57:15 [2023-11-11T14:57:15.206+00:00][WARN ][process] TimeoutError: Request timed out
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
2023-11-11 22:57:15     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-11-11 22:57:15     at runNextTicks (node:internal/process/task_queues:64:3)
2023-11-11 22:57:15     at listOnTimeout (node:internal/timers:538:9)
2023-11-11 22:57:15     at processTimers (node:internal/timers:512:7)
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
2023-11-11 22:57:15     at ClientTraced.GetApi [as get] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/get.js:36:12)
2023-11-11 22:57:15 [2023-11-11T14:57:15.228+00:00][ERROR][plugins.apm] Failed to add API keys to APM package policies
2023-11-11 22:57:15 [2023-11-11T14:57:15.229+00:00][ERROR][plugins.apm] TimeoutError: Request timed out
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
2023-11-11 22:57:15     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-11-11 22:57:15     at runNextTicks (node:internal/process/task_queues:64:3)
2023-11-11 22:57:15     at listOnTimeout (node:internal/timers:538:9)
2023-11-11 22:57:15     at processTimers (node:internal/timers:512:7)
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
2023-11-11 22:57:15     at ClientTraced.SearchApi [as search] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/search.js:66:12)
2023-11-11 22:57:15 [2023-11-11T14:57:15.401+00:00][ERROR][plugins.apm] Failed to schedule APM source map migration
2023-11-11 22:57:15 [2023-11-11T14:57:15.402+00:00][ERROR][plugins.apm] TimeoutError: Request timed out
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:527:31)
2023-11-11 22:57:15     at processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-11-11 22:57:15     at runNextTicks (node:internal/process/task_queues:64:3)
2023-11-11 22:57:15     at processTimers (node:internal/timers:509:9)
2023-11-11 22:57:15     at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
2023-11-11 22:57:15     at ClientTraced.CreateApi [as create] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/create.js:43:12)
2023-11-11 22:57:16 [2023-11-11T14:57:16.261+00:00][INFO ][plugins.fleet] Task Fleet-Usage-Logger-Task scheduled with interval 15m
2023-11-11 22:57:16 [2023-11-11T14:57:16.849+00:00][INFO ][status] Kibana is now degraded (was unavailable)
2023-11-11 22:57:18 [2023-11-11T14:57:18.110+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
2023-11-11 22:57:31 [2023-11-11T14:57:30.968+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Starting monitoring stats collection
2023-11-11 22:57:31 [2023-11-11T14:57:31.616+00:00][INFO ][plugins.fleet] Beginning fleet setup
2023-11-11 22:57:34 [2023-11-11T14:57:34.614+00:00][INFO ][plugins.telemetry] Telemetry collection is enabled. For more information on telemetry settings, refer to https://www.elastic.co/guide/en/kibana/8.11/telemetry-settings-kbn.html.
2023-11-11 22:57:35 [2023-11-11T14:57:35.228+00:00][INFO ][plugins.ruleRegistry] Installing ILM policy .preview.alerts-security.alerts-policy
2023-11-11 22:57:35 [2023-11-11T14:57:35.354+00:00][INFO ][plugins.observability] Installing SLO index template [.slo-observability.sli]
2023-11-11 22:57:36 [2023-11-11T14:57:36.181+00:00][INFO ][plugins.alerting] Installing component template .alerts-stack.alerts-mappings
2023-11-11 22:57:36 [2023-11-11T14:57:36.268+00:00][INFO ][plugins.alerting] Installing component template .alerts-observability.slo.alerts-mappings
2023-11-11 22:57:36 [2023-11-11T14:57:36.276+00:00][INFO ][plugins.alerting] Installing component template .alerts-observability.threshold.alerts-mappings
2023-11-11 22:57:36 [2023-11-11T14:57:36.305+00:00][INFO ][plugins.alerting] Installing component template .alerts-ml.anomaly-detection.alerts-mappings
2023-11-11 22:57:36 [2023-11-11T14:57:36.422+00:00][INFO ][plugins.alerting] Installing component template .alerts-observability.uptime.alerts-mappings
2023-11-11 22:57:36 [2023-11-11T14:57:36.429+00:00][INFO ][plugins.alerting] Installing component template .alerts-observability.logs.alerts-mappings
2023-11-11 22:57:36 [2023-11-11T14:57:36.435+00:00][INFO ][plugins.alerting] Installing component template .alerts-observability.metrics.alerts-mappings
2023-11-11 22:57:36 [2023-11-11T14:57:36.924+00:00][INFO ][plugins.alerting] Installing component template .alerts-security.alerts-mappings
2023-11-11 22:57:37 [2023-11-11T14:57:37.022+00:00][INFO ][plugins.alerting] Installing component template .alerts-observability.apm.alerts-mappings
2023-11-11 22:57:50 [2023-11-11T14:57:50.295+00:00][WARN ][plugins.kibanaUsageCollection] Average event loop delay threshold exceeded 350ms. Received 376.32139264ms. See https://ela.st/kibana-scaling-considerations for more information about scaling Kibana.
16

Hi Drive by comments.

First, I would always get Elasticsearch and Kibana working first before adding the other containers.

Second, I would work from the official compose.

And just reduce that down to a single node first. Get that all running before doing anything else.... I think I have a copy of that but I'm not at my desk right now.

Third, this has caused a number of issues with other people.

The setup only runs once because once the files are created it has logic to not run again. If for some reason it did not create the certs correctly, the correct certs will never be created again. The setup only runs the first time.

I would suggest since you haven't really got it running that you purge all the the volumes and try starting it over again.

Just a couple thoughts.

17

This is peculiar. I now can see the localhost:5601 webpage!

image
image1486×494 114 KB

But when I send a curl to 9200. I get empty request. When I open localhost:9200, I also dont get a response.

Is this intended behaviour?

=============================================
I would not be sure how to add additional containers. In the beginning, I start by trying to follow the single container arrangement but I kept failing/get stuck because

  1. I was setting up docker containers independent of each other. Have to manually start es and kib on sep clicks
  2. Cannot get ES and Kib to talk to e/o despite defining them on same network.
  3. I didnt know how docker-compose.yml come into the picture nor could I even find templates for it.

A lot of the official ES documentation just arbitrarily puts stuff in CLI format without actually telling clearly where or what to run that specific command, markup and etc. This makes it difficult for beginner as its ambigious which CLI to use - cmd? powershell? ubuntu? or smth else.

Sometimes we also dont even know what markup or language is used when something is written in code format.

PERFECT EG

image
image876×466 25.2 KB

I had to find YouTube video for easier visualisation.

Thus i resort to YouTube video on this multisetup arrangement where docker-compose.yml was used.

18

You need to use * https*.

Try curl -k https://localhost:9200 -u elastic:PASSWORD_YOU_USED_THE_ENV_FILE

Not sure what was your issue, I was able to run your docker-compose.yml without any issues.

19

Oh dear i have been using http coz Chat said so

image
image797×457 18.1 KB

Yes! Now I can get this finally!
image

Now I have a next part involving Logstash. I think i create a new Thread about it

20

Be careful using chat GPT for elasticsearch instructions as the data is several years old and we've had significant changes since then.

Also, what's worse sometimes as you just discovered the results can seem very correct and convincing and yet be incorrect.

1 Like