[2023-02-02T18:22:39,230][INFO ][o.e.e.NodeEnvironment ] [name] using [1] data paths, mounts [[/data (/dev/data)]], net usable_space [55.9gb], net total_space [98.4gb], types [ext4]
[2023-02-02T18:22:39,231][INFO ][o.e.e.NodeEnvironment ] [name] heap size [2gb], compressed ordinary object pointers [true]
[2023-02-02T18:22:39,352][INFO ][o.e.n.Node ] [name] node name [name], node ID [NSSQNNryRkeZsMxOuFd7gQ], cluster name [cluster_name], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2023-02-02T18:22:43,029][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [name] [controller/26337] [Main.cc@122] controller (64 bit): Version 7.16.3 (Build b71e71b93f3e4d) Copyright (c) 2021 Elasticsearch BV
[2023-02-02T18:22:43,529][INFO ][o.e.x.s.a.Realms ] [name] license mode is [trial], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-02-02T18:22:43,536][INFO ][o.e.x.s.a.s.FileRolesStore] [name] parsed [0] roles from file [/data/elasticsearch/config/roles.yml]
[2023-02-02T18:22:43,830][ERROR][o.e.b.Bootstrap ] [name] Exception
java.io.UncheckedIOException: Failed to create persistent cache writers
at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createWriters(PersistentCache.java:346) ~[?:?]
at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.<init>(PersistentCache.java:100) ~[?:?]
at org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots.createComponents(SearchableSnapshots.java:333) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:733) ~[elasticsearch-7.16.3.jar:7.16.3]
at java.util.stream.ReferencePipeline$7$1.accept(Unknown Source) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:747) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) [elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) [elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) [elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) [elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-7.16.3.jar:7.16.3]
at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) [elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.16.3.jar:7.16.3]
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by another program: /data/elasticsearch/data/nodes/0/snapshot_cache/write.lock
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:130) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:923) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createCacheIndexWriter(PersistentCache.java:377) ~[?:?]
at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createWriters(PersistentCache.java:342) ~[?:?]
... 22 more
[2023-02-02T18:22:43,835][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [name] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.io.UncheckedIOException: Failed to create persistent cache writers
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.3.jar:7.16.3]
at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.3.jar:7.16.3]
Caused by: java.io.UncheckedIOException: Failed to create persistent cache writers
at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createWriters(PersistentCache.java:346) ~[?:?]
at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.<init>(PersistentCache.java:100) ~[?:?]
at org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots.createComponents(SearchableSnapshots.java:333) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:733) ~[elasticsearch-7.16.3.jar:7.16.3]
at java.util.stream.ReferencePipeline$7$1.accept(Unknown Source) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:747) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.3.jar:7.16.3]
... 6 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by another program: /data/elasticsearch/data/nodes/0/snapshot_cache/write.lock
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:130) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:923) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createCacheIndexWriter(PersistentCache.java:377) ~[?:?]
at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createWriters(PersistentCache.java:342) ~[?:?]
at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.<init>(PersistentCache.java:100) ~[?:?]
at org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots.createComponents(SearchableSnapshots.java:333) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:733) ~[elasticsearch-7.16.3.jar:7.16.3]
at java.util.stream.ReferencePipeline$7$1.accept(Unknown Source) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:747) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.3.jar:7.16.3]
... 6 more
[2023-02-02T18:22:43,869][INFO ][o.e.x.m.p.NativeController] [name] Native controller process has stopped - no new native processes can be started
The first log I found is as above. I tried everything I found on Google to solve this problem, but it failed.
[2023-02-02T21:00:36,072][INFO ][o.e.e.NodeEnvironment ] [name] using [1] data paths, mounts [[/data (/dev/data)]], net usable_space [55.6gb], net total_space [98.4gb], types [ext4]
[2023-02-02T21:00:36,072][INFO ][o.e.e.NodeEnvironment ] [name] heap size [2gb], compressed ordinary object pointers [true]
[2023-02-02T21:00:36,116][INFO ][o.e.n.Node ] [name] node name [name], node ID [ljP7-iEISP2bDFseryxhhA], cluster name [cluster_name], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2023-02-02T21:00:39,781][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [name] [controller/30735] [Main.cc@122] controller (64 bit): Version 7.16.3 (Build b71e71b93f3e4d) Copyright (c) 2021 Elasticsearch BV
[2023-02-02T21:00:40,136][INFO ][o.e.x.s.a.Realms ] [name] license mode is [trial], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-02-02T21:00:40,145][INFO ][o.e.x.s.a.s.FileRolesStore] [name] parsed [0] roles from file [/data/elasticsearch/config/roles.yml]
[2023-02-02T21:00:40,690][INFO ][o.e.i.g.ConfigDatabases ] [name] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/data/elasticsearch/config/ingest-geoip] for changes
[2023-02-02T21:00:40,691][INFO ][o.e.i.g.DatabaseNodeService] [name] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-16478923630485953060/geoip-databases/ljP7-iEISP2bDFseryxhhA]
[2023-02-02T21:00:41,185][INFO ][o.e.t.NettyAllocator ] [name] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2023-02-02T21:00:41,241][INFO ][o.e.d.DiscoveryModule ] [name] using discovery type [zen] and seed hosts providers [settings]
[2023-02-02T21:00:41,647][INFO ][o.e.g.DanglingIndicesState] [name] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-02-02T21:00:42,081][INFO ][o.e.n.Node ] [name] initialized
[2023-02-02T21:00:42,082][INFO ][o.e.n.Node ] [name] starting ...
[2023-02-02T21:00:42,113][INFO ][o.e.x.s.c.f.PersistentCache] [name] persistent cache index loaded
[2023-02-02T21:00:42,115][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [name] deprecation component started
[2023-02-02T21:00:42,193][INFO ][o.e.t.TransportService ] [name] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2023-02-02T21:00:42,454][WARN ][o.e.b.BootstrapChecks ] [name] memory locking requested for elasticsearch process but memory is not locked
[2023-02-02T21:00:42,456][INFO ][o.e.c.c.Coordinator ] [name] cluster UUID [eB0SDrwlSP-905wyIowBFQ]
[2023-02-02T21:00:42,578][INFO ][o.e.c.s.MasterService ] [name] elected-as-master ([1] nodes joined)[{name}{ljP7-iEISP2bDFseryxhhA}{uuqbJScFQsWlGI-8wfv6qA}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 3, version: 63, delta: master node changed {previous [], current [{name}{ljP7-iEISP2bDFseryxhhA}{uuqbJScFQsWlGI-8wfv6qA}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}
[2023-02-02T21:00:42,650][INFO ][o.e.c.s.ClusterApplierService] [name] master node changed {previous [], current [{name}{ljP7-iEISP2bDFseryxhhA}{uuqbJScFQsWlGI-8wfv6qA}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}, term: 3, version: 63, reason: Publication{term=3, version=63}
[2023-02-02T21:00:42,698][INFO ][o.e.h.AbstractHttpServerTransport] [name] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2023-02-02T21:00:42,699][INFO ][o.e.n.Node ] [name] started
[2023-02-02T21:00:42,971][INFO ][o.e.l.LicenseService ] [name] license [9903a757-0105-4659-8056-6974082ed965] mode [basic] - valid
[2023-02-02T21:00:42,972][INFO ][o.e.x.s.a.Realms ] [name] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-02-02T21:00:42,973][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [name] Active license is now [BASIC]; Security is enabled
[2023-02-02T21:00:42,976][INFO ][o.e.g.GatewayService ] [name] recovered [2] indices into cluster_state
[2023-02-02T21:00:43,346][INFO ][o.e.c.r.a.AllocationService] [name] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.ds-.logs-deprecation.elasticsearch-default-2023.02.02-000001][0], [.ds-ilm-history-5-2023.02.02-000001][0]]]).
Afterwards, I deleted and reinstalled the Elasticsearch, and the logs I found were as follows.
It also tried everything, but it didn't succeed.
Thereafter, two logs are generated alternately.
I wanted to replace the hard disk, but I couldn't because I couldn't do that.
Please let me know if the hard disk is a problem or if there is another solution.