Hi,
I am trying to setup Enterprise Search through Docker compose using the guide here - Run Enterprise Search server using Docker images | Elastic Enterprise Search documentation [8.4] | Elastic.
While Kibana container can startup correctly and stay running, both Elasticsearch and Enterpricsesearch will start and and stop running.
Please note, if I start only the Elasticsearch, it stays running and will not stop. However once I start Kibana in less than 2 minutes Elasticsearch will terinate itself as a result of this I cannot run Enterprisesearch.
Elasticsearch Logs
{"@timestamp":"2022-10-21T19:06:56.788Z", "log.level":"ERROR", "message":"exception during geoip databases update", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#3]","log.logger":"org.elasticsearch.ingest.geoip.GeoIpDownloader","elasticsearch.cluster.uuid":"1gN-FTxMSZqMVOvLR4m-Ig","elasticsearch.node.id":"pJ9pG8JJRPeJw8XTvHzDNA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-cluster","error.type":"org.elasticsearch.ElasticsearchException","error.message":"not all primary shards of [.geoip_databases] index are active","error.stack_trace":"org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active\n\tat org.elasticsearch.ingest.geoip@8.4.3/org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:134)\n\tat org.elasticsearch.ingest.geoip@8.4.3/org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:274)\n\tat org.elasticsearch.ingest.geoip@8.4.3/org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:102)\n\tat org.elasticsearch.ingest.geoip@8.4.3/org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:48)\n\tat org.elasticsearch.server@8.4.3/org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42)\n\tat org.elasticsearch.server@8.4.3/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:769)\n\tat org.elasticsearch.server@8.4.3/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"}
{"@timestamp":"2022-10-21T19:06:57.360Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#3]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"1gN-FTxMSZqMVOvLR4m-Ig","elasticsearch.node.id":"pJ9pG8JJRPeJw8XTvHzDNA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-cluster"}
{"@timestamp":"2022-10-21T19:06:57.393Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-ASN.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"1gN-FTxMSZqMVOvLR4m-Ig","elasticsearch.node.id":"pJ9pG8JJRPeJw8XTvHzDNA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-cluster"}
{"@timestamp":"2022-10-21T19:06:57.393Z", "log.level": "INFO", "current.health":"GREEN","message":"Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.kibana-event-log-8.4.3-000001][0]]]).","previous.health":"RED","reason":"shards started [[.kibana-event-log-8.4.3-000001][0]]" , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"1gN-FTxMSZqMVOvLR4m-Ig","elasticsearch.node.id":"pJ9pG8JJRPeJw8XTvHzDNA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-cluster"}
{"@timestamp":"2022-10-21T19:06:57.992Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#2]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"1gN-FTxMSZqMVOvLR4m-Ig","elasticsearch.node.id":"pJ9pG8JJRPeJw8XTvHzDNA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-cluster"}
ERROR: Elasticsearch exited unexpectedly
{"@timestamp":"2022-10-21T19:06:58.718Z", "log.level": "WARN", "message":"http client did not trust this server's certificate, closing connection Netty4HttpChannel{localAddress=/172.18.0.2:9200, remoteAddress=/172.18.0.1:35986}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#10]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"1gN-FTxMSZqMVOvLR4m-Ig","elasticsearch.node.id":"pJ9pG8JJRPeJw8XTvHzDNA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-cluster"}
{"@timestamp":"2022-10-21T19:06:58.723Z", "log.level": "WARN", "message":"http client did not trust this server's certificate, closing connection Netty4HttpChannel{localAddress=/172.18.0.2:9200, remoteAddress=/172.18.0.1:35990}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][transport_worker][T#11]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"1gN-FTxMSZqMVOvLR4m-Ig","elasticsearch.node.id":"pJ9pG8JJRPeJw8XTvHzDNA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-cluster"}
{"@timestamp":"2022-10-21T19:07:50.014Z", "log.level": "INFO", "message":"256 finished with response BulkByScrollResponse[took=112.4ms,timed_out=false,sliceId=null,updated=25,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][system_write][T#3]","log.logger":"org.elasticsearch.tasks.LoggingTaskListener","trace.id":"6ab1762f313938b9abfb932b9d3baf76","elasticsearch.cluster.uuid":"1gN-FTxMSZqMVOvLR4m-Ig","elasticsearch.node.id":"pJ9pG8JJRPeJw8XTvHzDNA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-cluster"}
{"@timestamp":"2022-10-21T19:07:50.168Z", "log.level": "INFO", "message":"263 finished with response BulkByScrollResponse[took=238.4ms,timed_out=false,sliceId=null,updated=167,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][system_write][T#4]","log.logger":"org.elasticsearch.tasks.LoggingTaskListener","trace.id":"6ab1762f313938b9abfb932b9d3baf76","elasticsearch.cluster.uuid":"1gN-FTxMSZqMVOvLR4m-Ig","elasticsearch.node.id":"pJ9pG8JJRPeJw8XTvHzDNA","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"es-cluster"}
Kibana Logs
[2022-10-21T19:07:50.238+00:00][INFO ][plugins-system.standard] Starting [121] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,newsfeed,fieldFormats,expressions,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,lists,encryptedSavedObjects,cloud,snapshotRestore,screenshotting,telemetry,licenseManagement,eventLog,actions,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,unifiedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,eventAnnotation,dataViewFieldEditor,triggersActionsUi,transform,stackAlerts,ruleRegistry,discover,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,aiops,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,osquery,maps,dataVisualizer,ml,cases,timelines,sessionView,kubernetesSecurity,securitySolution,visTypeGauge,sharedUX,observability,synthetics,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,dataViewManagement]
[2022-10-21T19:07:51.364+00:00][INFO ][plugins.monitoring.monitoring] config sourced from: production cluster
[2022-10-21T19:07:52.576+00:00][INFO ][http.server.Kibana] http server running at http://0.0.0.0:5601
[2022-10-21T19:07:52.664+00:00][INFO ][status] Kibana is now degraded
[2022-10-21T19:07:52.789+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Starting monitoring stats collection
[2022-10-21T19:07:52.789+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2022-10-21T19:07:52.804+00:00][INFO ][plugins.ruleRegistry] Installed common resources shared between all indices
[2022-10-21T19:07:52.804+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-security.alerts
[2022-10-21T19:07:52.806+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .preview.alerts-security.alerts
[2022-10-21T19:07:52.807+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.uptime.alerts
[2022-10-21T19:07:52.808+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.logs.alerts
[2022-10-21T19:07:52.809+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.metrics.alerts
[2022-10-21T19:07:52.810+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.apm.alerts
[2022-10-21T19:07:52.840+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-security.alerts
[2022-10-21T19:07:52.852+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.metrics.alerts
[2022-10-21T19:07:52.854+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.apm.alerts
[2022-10-21T19:07:52.855+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.logs.alerts
[2022-10-21T19:07:52.856+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.uptime.alerts
[2022-10-21T19:07:52.872+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .preview.alerts-security.alerts
[2022-10-21T19:07:53.138+00:00][INFO ][plugins.ml] Task ML:saved-objects-sync-task: scheduled with interval 1h
[2022-10-21T19:07:53.447+00:00][INFO ][plugins.fleet] Fleet setup completed
[2022-10-21T19:07:53.458+00:00][INFO ][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask
[2022-10-21T19:07:53.839+00:00][INFO ][plugins.synthetics] Installed synthetics index templates
[2022-10-21T19:07:58.717+00:00][INFO ][plugins.ml] Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization
[2022-10-21T19:08:01.680+00:00][INFO ][status] Kibana is now available (was degraded)
[2022-10-21T19:09:11.670+00:00][ERROR][plugins.taskManager] Failed to poll for work: ConnectionError: connect ECONNREFUSED 127.0.0.1:9200
[2022-10-21T19:09:11.675+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 127.0.0.1:9200
[2022-10-21T19:09:11.768+00:00][INFO ][status] Kibana is now critical (was available)
[2022-10-21T19:09:19.695+00:00][WARN ][plugins.usageCollection.usage-collection.usage-counters-service] ConnectionError: connect ECONNREFUSED 127.0.0.1:9200
at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:525:31)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
[2022-10-21T19:09:19.784+00:00][WARN ][plugins.usageCollection.usage-collection.usage-counters-service] ConnectionError: connect ECONNREFUSED 127.0.0.1:9200
at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:525:31)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
at ClientTraced.UpdateApi [as update] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/update.js:50:12)
[2022-10-21T19:09:20.803+00:00][WARN ][plugins.monitoring.monitoring.kibana-monitoring] ConnectionError: connect ECONNREFUSED 127.0.0.1:9200
at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:525:31)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
at Monitoring.bulk (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/monitoring.js:53:16)
at sendBulkPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/lib/send_bulk_payload.js:21:10)
at BulkUploader._onPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:187:12)
at BulkUploader._fetchAndUpload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:173:9)
[2022-10-21T19:09:20.803+00:00][WARN ][plugins.monitoring.monitoring.kibana-monitoring] Unable to bulk upload the stats payload to the local cluster
[2022-10-21T19:09:27.651+00:00][WARN ][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 127.0.0.1:9200 error
[2022-10-21T19:09:27.652+00:00][INFO ][plugins.searchprofiler] You cannot use searchprofiler because license information is not available at this time.
[2022-10-21T19:09:27.653+00:00][INFO ][plugins.painlessLab] You cannot use painlessLab because license information is not available at this time.
[2022-10-21T19:09:27.653+00:00][INFO ][plugins.snapshotRestore] You cannot use snapshot_restore because license information is not available at this time.
[2022-10-21T19:09:27.654+00:00][INFO ][plugins.transform] You cannot use transform because license information is not available at this time.
[2022-10-21T19:09:27.654+00:00][INFO ][plugins.remoteClusters] You cannot use Remote Clusters because license information is not available at this time.
[2022-10-21T19:09:27.654+00:00][INFO ][plugins.indexLifecycleManagement] You cannot use index_lifecycle_management because license information is not available at this time.
[2022-10-21T19:09:27.654+00:00][INFO ][plugins.rollup] You cannot use rollup because license information is not available at this time.
[2022-10-21T19:09:27.658+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped
[2022-10-21T19:09:29.670+00:00][WARN ][plugins.securitySolution] Unable to verify endpoint policies in line with license change: failed to fetch package policies: connect ECONNREFUSED 127.0.0.1:9200
[2022-10-21T19:09:30.687+00:00][WARN ][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 127.0.0.1:9200 error
[2022-10-21T19:09:30.687+00:00][WARN ][plugins.monitoring.monitoring] X-Pack Monitoring Cluster Alerts will not be available: connect ECONNREFUSED 127.0.0.1:9200
[2022-10-21T19:09:31.259+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2022-10-21T19:09:31.701+00:00][WARN ][plugins.monitoring.monitoring.kibana-monitoring] ConnectionError: connect ECONNREFUSED 127.0.0.1:9200
at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:525:31)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/create_transport.js:58:16)
at Monitoring.bulk (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/monitoring.js:53:16)
at sendBulkPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/lib/send_bulk_payload.js:21:10)
at BulkUploader._onPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:187:12)
at BulkUploader._fetchAndUpload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:173:9)
[2022-10-21T19:09:31.701+00:00][WARN ][plugins.monitoring.monitoring.kibana-monitoring] Unable to bulk upload the stats payload to the local cluster
[2022-10-21T19:09:35.635+00:00][WARN ][plugins.licensing] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 127.0.0.1:9200 error
Thanks in anticipation for your help.
Kamal