I am using docker sebp/elk to build my elk environment. normally it runs well, but several days ago, the PC is restarted and when I want to restart the docker, it 99% fail and 1% run successfully. and after inspecting the docker instance, the ExitCode=1. here is the log from elasticsearch.
* Starting periodic command scheduler cron
...done.
* Starting Elasticsearch Server
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.elasticsearch.bootstrap.Elasticsearch (file:/opt/elasticsearch/lib/elasticsearch-7.15.2.jar)
WARNING: Please consider reporting this to the maintainers of org.elasticsearch.bootstrap.Elasticsearch
WARNING: System::setSecurityManager will be removed in a future release
...done.
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.elasticsearch.bootstrap.Security (file:/opt/elasticsearch/lib/elasticsearch-7.15.2.jar)
WARNING: Please consider reporting this to the maintainers of org.elasticsearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
waiting for Elasticsearch to be up (1/30)
waiting for Elasticsearch to be up (2/30)
waiting for Elasticsearch to be up (3/30)
waiting for Elasticsearch to be up (4/30)
waiting for Elasticsearch to be up (5/30)
waiting for Elasticsearch to be up (6/30)
waiting for Elasticsearch to be up (7/30)
waiting for Elasticsearch to be up (8/30)
waiting for Elasticsearch to be up (9/30)
waiting for Elasticsearch to be up (10/30)
waiting for Elasticsearch to be up (11/30)
waiting for Elasticsearch to be up (12/30)
waiting for Elasticsearch to be up (13/30)
waiting for Elasticsearch to be up (14/30)
waiting for Elasticsearch to be up (15/30)
waiting for Elasticsearch to be up (16/30)
waiting for Elasticsearch to be up (17/30)
waiting for Elasticsearch to be up (18/30)
waiting for Elasticsearch to be up (19/30)
waiting for Elasticsearch to be up (20/30)
waiting for Elasticsearch to be up (21/30)
waiting for Elasticsearch to be up (22/30)
waiting for Elasticsearch to be up (23/30)
waiting for Elasticsearch to be up (24/30)
waiting for Elasticsearch to be up (25/30)
waiting for Elasticsearch to be up (26/30)
waiting for Elasticsearch to be up (27/30)
waiting for Elasticsearch to be up (28/30)
waiting for Elasticsearch to be up (29/30)
waiting for Elasticsearch to be up (30/30)
Couldn't start Elasticsearch. Exiting.
Elasticsearch log follows below.
[2023-08-03T09:14:59,937][INFO ][o.e.n.Node ] [elk] version[7.15.2], pid[209], build[default/tar/93d5a7f6192e8a1a12e154a2b81bf6fa7309da0c/2021-11-04T14:04:42.515624022Z], OS[Linux/4.4.0-173-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.1/17.0.1+12]
[2023-08-03T09:15:00,038][INFO ][o.e.n.Node ] [elk] JVM home [/opt/elasticsearch/jdk], using bundled JDK [true]
[2023-08-03T09:15:00,039][INFO ][o.e.n.Node ] [elk] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-4268245752759272744, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms31744m, -Xmx31744m, -XX:MaxDirectMemorySize=16642998272, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/opt/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2023-08-03T09:15:06,731][INFO ][o.e.p.PluginsService ] [elk] loaded module [aggs-matrix-stats]
[2023-08-03T09:15:06,732][INFO ][o.e.p.PluginsService ] [elk] loaded module [analysis-common]
[2023-08-03T09:15:06,732][INFO ][o.e.p.PluginsService ] [elk] loaded module [constant-keyword]
[2023-08-03T09:15:06,733][INFO ][o.e.p.PluginsService ] [elk] loaded module [frozen-indices]
[2023-08-03T09:15:06,733][INFO ][o.e.p.PluginsService ] [elk] loaded module [ingest-common]
[2023-08-03T09:15:06,734][INFO ][o.e.p.PluginsService ] [elk] loaded module [ingest-geoip]
[2023-08-03T09:15:06,734][INFO ][o.e.p.PluginsService ] [elk] loaded module [ingest-user-agent]
[2023-08-03T09:15:06,738][INFO ][o.e.p.PluginsService ] [elk] loaded module [kibana]
[2023-08-03T09:15:06,739][INFO ][o.e.p.PluginsService ] [elk] loaded module [lang-expression]
[2023-08-03T09:15:06,739][INFO ][o.e.p.PluginsService ] [elk] loaded module [lang-mustache]
[2023-08-03T09:15:06,740][INFO ][o.e.p.PluginsService ] [elk] loaded module [lang-painless]
[2023-08-03T09:15:06,740][INFO ][o.e.p.PluginsService ] [elk] loaded module [mapper-extras]
[2023-08-03T09:15:06,741][INFO ][o.e.p.PluginsService ] [elk] loaded module [mapper-version]
[2023-08-03T09:15:06,741][INFO ][o.e.p.PluginsService ] [elk] loaded module [parent-join]
[2023-08-03T09:15:06,742][INFO ][o.e.p.PluginsService ] [elk] loaded module [percolator]
[2023-08-03T09:15:06,742][INFO ][o.e.p.PluginsService ] [elk] loaded module [rank-eval]
[2023-08-03T09:15:06,742][INFO ][o.e.p.PluginsService ] [elk] loaded module [reindex]
[2023-08-03T09:15:06,743][INFO ][o.e.p.PluginsService ] [elk] loaded module [repositories-metering-api]
[2023-08-03T09:15:06,743][INFO ][o.e.p.PluginsService ] [elk] loaded module [repository-encrypted]
[2023-08-03T09:15:06,744][INFO ][o.e.p.PluginsService ] [elk] loaded module [repository-url]
[2023-08-03T09:15:06,744][INFO ][o.e.p.PluginsService ] [elk] loaded module [runtime-fields-common]
[2023-08-03T09:15:06,744][INFO ][o.e.p.PluginsService ] [elk] loaded module [search-business-rules]
[2023-08-03T09:15:06,745][INFO ][o.e.p.PluginsService ] [elk] loaded module [searchable-snapshots]
[2023-08-03T09:15:06,745][INFO ][o.e.p.PluginsService ] [elk] loaded module [snapshot-repo-test-kit]
[2023-08-03T09:15:06,746][INFO ][o.e.p.PluginsService ] [elk] loaded module [spatial]
[2023-08-03T09:15:06,746][INFO ][o.e.p.PluginsService ] [elk] loaded module [transform]
[2023-08-03T09:15:06,747][INFO ][o.e.p.PluginsService ] [elk] loaded module [transport-netty4]
[2023-08-03T09:15:06,747][INFO ][o.e.p.PluginsService ] [elk] loaded module [unsigned-long]
[2023-08-03T09:15:06,747][INFO ][o.e.p.PluginsService ] [elk] loaded module [vector-tile]
[2023-08-03T09:15:06,748][INFO ][o.e.p.PluginsService ] [elk] loaded module [vectors]
[2023-08-03T09:15:06,748][INFO ][o.e.p.PluginsService ] [elk] loaded module [wildcard]
[2023-08-03T09:15:06,749][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-aggregate-metric]
[2023-08-03T09:15:06,749][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-analytics]
[2023-08-03T09:15:06,749][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-async]
[2023-08-03T09:15:06,750][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-async-search]
[2023-08-03T09:15:06,750][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-autoscaling]
[2023-08-03T09:15:06,751][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-ccr]
[2023-08-03T09:15:06,751][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-core]
[2023-08-03T09:15:06,751][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-data-streams]
[2023-08-03T09:15:06,752][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-deprecation]
[2023-08-03T09:15:06,753][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-enrich]
[2023-08-03T09:15:06,754][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-eql]
[2023-08-03T09:15:06,754][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-fleet]
[2023-08-03T09:15:06,754][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-graph]
[2023-08-03T09:15:06,755][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-identity-provider]
[2023-08-03T09:15:06,755][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-ilm]
[2023-08-03T09:15:06,760][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-logstash]
[2023-08-03T09:15:06,760][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-ml]
[2023-08-03T09:15:06,761][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-monitoring]
[2023-08-03T09:15:06,761][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-ql]
[2023-08-03T09:15:06,762][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-rollup]
[2023-08-03T09:15:06,762][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-security]
[2023-08-03T09:15:06,762][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-shutdown]
[2023-08-03T09:15:06,763][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-sql]
[2023-08-03T09:15:06,765][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-stack]
[2023-08-03T09:15:06,766][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-text-structure]
[2023-08-03T09:15:06,766][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-voting-only-node]
[2023-08-03T09:15:06,770][INFO ][o.e.p.PluginsService ] [elk] loaded module [x-pack-watcher]
[2023-08-03T09:15:06,771][INFO ][o.e.p.PluginsService ] [elk] no plugins loaded
[2023-08-03T09:15:06,880][INFO ][o.e.e.NodeEnvironment ] [elk] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/sda3)]], net usable_space [2.3tb], net total_space [14.1tb], types [ext4]
[2023-08-03T09:15:06,881][INFO ][o.e.e.NodeEnvironment ] [elk] heap size [31gb], compressed ordinary object pointers [true]
[2023-08-03T09:15:08,184][INFO ][o.e.n.Node ] [elk] node name [elk], node ID [uJ6iZBqkTZyYw-OLsGDIDw], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2023-08-03T09:15:21,660][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elk] [controller/269] [Main.cc@122] controller (64 bit): Version 7.15.2 (Build 65497bb5299534) Copyright (c) 2021 Elasticsearch BV
[2023-08-03T09:15:24,742][INFO ][o.e.i.g.LocalDatabases ] [elk] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/etc/elasticsearch/ingest-geoip] for changes
[2023-08-03T09:15:24,759][INFO ][o.e.i.g.DatabaseRegistry ] [elk] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-4268245752759272744/geoip-databases/uJ6iZBqkTZyYw-OLsGDIDw]
[2023-08-03T09:15:26,443][INFO ][o.e.t.NettyAllocator ] [elk] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]
[2023-08-03T09:15:27,791][INFO ][o.e.d.DiscoveryModule ] [elk] using discovery type [zen] and seed hosts providers [settings]
[2023-08-03T09:15:29,121][INFO ][o.e.g.DanglingIndicesState] [elk] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-08-03T09:15:30,200][INFO ][o.e.n.Node ] [elk] initialized
[2023-08-03T09:15:30,201][INFO ][o.e.n.Node ] [elk] starting ...
[2023-08-03T09:15:30,566][INFO ][o.e.x.s.c.f.PersistentCache] [elk] persistent cache index loaded
and here is the docker inspect result:
[
{
"Id": "51d1d1679ed8c0d4ff5a4bc0a595aa54c1089e48a90b779f362f797699d3ca2b",
"Created": "2023-08-03T09:14:43.481031418Z",
"Path": "/usr/local/bin/start.sh",
"Args": [],
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 1,
"Error": "",
"StartedAt": "2023-08-03T09:14:45.524443388Z",
"FinishedAt": "2023-08-03T09:15:34.572194175Z"
},
"Image": "sha256:2b1eae925a590b2f0ad0173f11fdd71f69d716bb99c7149bc1e3ba90c65e4e62",
"ResolvConfPath": "/var/lib/docker/containers/51d1d1679ed8c0d4ff5a4bc0a595aa54c1089e48a90b779f362f797699d3ca2b/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/51d1d1679ed8c0d4ff5a4bc0a595aa54c1089e48a90b779f362f797699d3ca2b/hostname",
"HostsPath": "/var/lib/docker/containers/51d1d1679ed8c0d4ff5a4bc0a595aa54c1089e48a90b779f362f797699d3ca2b/hosts",
"LogPath": "/var/lib/docker/containers/51d1d1679ed8c0d4ff5a4bc0a595aa54c1089e48a90b779f362f797699d3ca2b/51d1d1679ed8c0d4ff5a4bc0a595aa54c1089e48a90b779f362f797699d3ca2b-json.log",
"Name": "/elk-formal"
...