I Run Elasticsearch on docker production on aws ec2(ubuntu).It was working fine.my elasticsearch got exited today after I updated an index I started again manually.after this the status of the containers changed to unhealthy
Why this happens?What can I do to recover the cluster?
Here is the logs of it:
es01:
amp":"2022-09-04T01:38:32.025Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#3]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"IKg4rU8uR9WW9D6yPqfEZQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-04T01:38:35.854Z", "log.level": "INFO", "message":"evicted [0] entries from cache after reloading database [/tmp/elasticsearch-1445181459173855649/geoip-databases/IKg4rU8uR9WW9D6yPqfEZQ/GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"IKg4rU8uR9WW9D6yPqfEZQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-04T01:38:35.854Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"IKg4rU8uR9WW9D6yPqfEZQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:33.554Z", "log.level": "INFO", "message":"evicted [0] entries from cache after reloading database [/tmp/elasticsearch-1445181459173855649/geoip-databases/IKg4rU8uR9WW9D6yPqfEZQ/GeoLite2-ASN.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"IKg4rU8uR9WW9D6yPqfEZQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:33.555Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-ASN.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"IKg4rU8uR9WW9D6yPqfEZQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:42.577Z", "log.level": "INFO", "message":"evicted [0] entries from cache after reloading database [/tmp/elasticsearch-1445181459173855649/geoip-databases/IKg4rU8uR9WW9D6yPqfEZQ/GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#4]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"IKg4rU8uR9WW9D6yPqfEZQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:42.577Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#4]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"IKg4rU8uR9WW9D6yPqfEZQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:46.151Z", "log.level": "INFO", "message":"evicted [0] entries from cache after reloading database [/tmp/elasticsearch-1445181459173855649/geoip-databases/IKg4rU8uR9WW9D6yPqfEZQ/GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"IKg4rU8uR9WW9D6yPqfEZQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:46.152Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][generic][T#1]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"IKg4rU8uR9WW9D6yPqfEZQ","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
ERROR: Elasticsearch exited unexpectedly
es02:
{"@timestamp":"2022-09-07T00:06:04.074Z", "log.level": "INFO", "current.health":"GREEN","message":"Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[restaurant_location][0]]]).","previous.health":"YELLOW","reason":"shards started [[restaurant_location][0]]" , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T00:07:50.954Z", "log.level": "INFO", "message":"[restaurant_location/cQuHs6RHSxqULZtzHDt7Aw] create_mapping", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T00:07:51.015Z", "log.level": "INFO", "message":"[restaurant_location/cQuHs6RHSxqULZtzHDt7Aw] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T00:07:51.066Z", "log.level": "INFO", "message":"[restaurant_location/cQuHs6RHSxqULZtzHDt7Aw] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T00:07:51.110Z", "log.level": "INFO", "message":"[restaurant_location/cQuHs6RHSxqULZtzHDt7Aw] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T00:07:51.222Z", "log.level": "INFO", "message":"[restaurant_location/cQuHs6RHSxqULZtzHDt7Aw] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T00:07:51.372Z", "log.level": "INFO", "message":"[restaurant_location/cQuHs6RHSxqULZtzHDt7Aw] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T00:07:51.769Z", "log.level": "INFO", "message":"[restaurant_location/cQuHs6RHSxqULZtzHDt7Aw] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:30:00.000Z", "log.level": "INFO", "message":"starting SLM retention snapshot cleanup task", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][trigger_engine_scheduler][T#1]","log.logger":"org.elasticsearch.xpack.slm.SnapshotRetentionTask","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:30:00.000Z", "log.level": "INFO", "message":"there are no repositories to fetch, SLM retention snapshot cleanup task complete", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][trigger_engine_scheduler][T#1]","log.logger":"org.elasticsearch.xpack.slm.SnapshotRetentionTask","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:00.001Z", "log.level": "INFO", "message":"triggering scheduled [ML] maintenance tasks", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#3]","log.logger":"org.elasticsearch.xpack.ml.MlDailyMaintenanceService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:00.002Z", "log.level": "INFO", "message":"Deleting expired data", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#3]","log.logger":"org.elasticsearch.xpack.ml.action.TransportDeleteExpiredDataAction","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:00.004Z", "log.level": "INFO", "message":"Successfully deleted [0] unused stats documents", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#2]","log.logger":"org.elasticsearch.xpack.ml.job.retention.UnusedStatsRemover","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:00.005Z", "log.level": "INFO", "message":"Completed deletion of expired ML data", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][ml_utility][T#3]","log.logger":"org.elasticsearch.xpack.ml.action.TransportDeleteExpiredDataAction","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:00.005Z", "log.level": "INFO", "message":"Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][ml_utility][T#3]","log.logger":"org.elasticsearch.xpack.ml.MlDailyMaintenanceService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:33.460Z", "log.level": "INFO", "message":"evicted [0] entries from cache after reloading database [/tmp/elasticsearch-16016688957705358733/geoip-databases/_On0JHF6SsCi8T2qU4FyFQ/GeoLite2-ASN.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#2]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:33.460Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-ASN.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#2]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:42.430Z", "log.level": "INFO", "message":"evicted [0] entries from cache after reloading database [/tmp/elasticsearch-16016688957705358733/geoip-databases/_On0JHF6SsCi8T2qU4FyFQ/GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#3]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:42.430Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#3]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:44.931Z", "log.level": "INFO", "message":"evicted [0] entries from cache after reloading database [/tmp/elasticsearch-16016688957705358733/geoip-databases/_On0JHF6SsCi8T2qU4FyFQ/GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#4]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T01:38:44.932Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#4]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:21:46.540Z", "log.level": "INFO", "message":"[restaurant_location/cQuHs6RHSxqULZtzHDt7Aw] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:26:25.772Z", "log.level": "INFO", "message":"transport connection to [{es01}{IKg4rU8uR9WW9D6yPqfEZQ}{zsLfR_mvRTuFDtvFwq6Ddg}{es01}{172.19.0.3}{172.19.0.3:9300}{cdfhilmrstw}] closed by remote", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][transport_worker][T#2]","log.logger":"org.elasticsearch.transport.ClusterConnectionManager","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:26:25.782Z", "log.level": "INFO", "current.health":"YELLOW","message":"Cluster health status changed from [GREEN] to [YELLOW] (reason: [{es01}{IKg4rU8uR9WW9D6yPqfEZQ}{zsLfR_mvRTuFDtvFwq6Ddg}{es01}{172.19.0.3}{172.19.0.3:9300}{cdfhilmrstw} reason: disconnected]).","previous.health":"GREEN","reason":"{es01}{IKg4rU8uR9WW9D6yPqfEZQ}{zsLfR_mvRTuFDtvFwq6Ddg}{es01}{172.19.0.3}{172.19.0.3:9300}{cdfhilmrstw} reason: disconnected" , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:26:25.802Z", "log.level": "INFO", "message":"node-left[{es01}{IKg4rU8uR9WW9D6yPqfEZQ}{zsLfR_mvRTuFDtvFwq6Ddg}{es01}{172.19.0.3}{172.19.0.3:9300}{cdfhilmrstw} reason: disconnected], term: 20, version: 1172, delta: removed {{es01}{IKg4rU8uR9WW9D6yPqfEZQ}{zsLfR_mvRTuFDtvFwq6Ddg}{es01}{172.19.0.3}{172.19.0.3:9300}{cdfhilmrstw}}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:26:25.906Z", "log.level": "INFO", "message":"removed {{es01}{IKg4rU8uR9WW9D6yPqfEZQ}{zsLfR_mvRTuFDtvFwq6Ddg}{es01}{172.19.0.3}{172.19.0.3:9300}{cdfhilmrstw}}, term: 20, version: 1172, reason: Publication{term=20, version=1172}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.ClusterApplierService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:26:25.950Z", "log.level": "INFO", "message":"scheduling reroute for delayed shards in [59.8s] (15 delayed shards)", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.DelayedAllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:26:25.956Z", "log.level": "INFO", "message":"primary-replica resync completed with 0 operations", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#6]","log.logger":"org.elasticsearch.index.shard.IndexShard","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster","tags":[" [.kibana-event-log-8.3.2-000002][0]"]}
{"@timestamp":"2022-09-07T17:26:25.961Z", "log.level": "INFO", "message":"primary-replica resync completed with 0 operations", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#3]","log.logger":"org.elasticsearch.index.shard.IndexShard","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster","tags":[" [.apm-custom-link][0]"]}
{"@timestamp":"2022-09-07T17:26:25.973Z", "log.level": "INFO", "message":"primary-replica resync completed with 0 operations", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][generic][T#6]","log.logger":"org.elasticsearch.index.shard.IndexShard","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster","tags":[" [restaurant_items][0]"]}
{"@timestamp":"2022-09-07T17:27:26.004Z", "log.level": "WARN", "message":"[.security-7][0] marking unavailable shards as stale: [iX31xcNqRSWD9UQRj0KbxA]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:27:26.086Z", "log.level": "WARN", "message":"[.kibana_task_manager_8.3.2_001][0] marking unavailable shards as stale: [oSWzHXHEQbWSh6bxZ7bf2g]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:27:26.335Z", "log.level": "WARN", "message":"[.apm-custom-link][0] marking unavailable shards as stale: [nBUe7MqYQGiNRDOWF4xDyw]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:27:26.385Z", "log.level": "WARN", "message":"[.async-search][0] marking unavailable shards as stale: [6uBmF5DGQTGCR1ZRD2Jj6w]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:27:26.445Z", "log.level": "WARN", "message":"[.tasks][0] marking unavailable shards as stale: [58f1y6RrSNOL3SccPyeyyQ]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:27:26.767Z", "log.level": "WARN", "message":"[restaurants][0] marking unavailable shards as stale: [0s-uHGuxSq2yDSbR_VEu8w]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:27:26.937Z", "log.level": "WARN", "message":"[.ds-.logs-deprecation.elasticsearch-default-2022.08.12-000002][0] marking unavailable shards as stale: [6woD3_CvRAqD8UKk6R9wzQ]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:27:27.278Z", "log.level": "WARN", "message":"[.kibana-event-log-8.3.2-000002][0] marking unavailable shards as stale: [vfw_hUhLRhWXVSviWg7lfA]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:27:28.657Z", "log.level": "WARN", "message":"[smfood-recipe-combo][0] marking unavailable shards as stale: [-XYM89GVS_yzcnpdncNGyQ]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:27:28.711Z", "log.level": "WARN", "message":"[smfood][0] marking unavailable shards as stale: [5J_Tpy1pQU6Upfacp4C8ZA]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:27:29.529Z", "log.level": "WARN", "message":"[.ds-.logs-deprecation.elasticsearch-default-2022.07.13-000001][0] marking unavailable shards as stale: [AhokeX6RTCKENYN2-qT6NQ]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:27:30.267Z", "log.level": "WARN", "message":"[.kibana-event-log-8.3.2-000001][0] marking unavailable shards as stale: [r-ZZmzgKR4SLnVZOEX4Zhg]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:27:33.246Z", "log.level": "WARN", "message":"[restaurant_location][0] marking unavailable shards as stale: [85FegeDxT5aLuTt7gR8CAA]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:27:34.940Z", "log.level": "WARN", "message":"[restaurant_items][0] marking unavailable shards as stale: [BQ6m973VSHmjz2z3eCp3-w]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-07T17:27:43.962Z", "log.level": "WARN", "message":"[cpg][0] marking unavailable shards as stale: [y6jmNn6aSEKQmYyHehWybA]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es02][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"lL9JIvXISd2QDpkbjdyMRQ","elasticsearch.node.id":"_On0JHF6SsCi8T2qU4FyFQ","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}```
ERROR: Elasticsearch exited unexpectedly
is there any solution for this?