# Elastic search does not work during operation, causing problems in restarting

**URL:** https://discuss.elastic.co/t/elastic-search-does-not-work-during-operation-causing-problems-in-restarting/324556
**Category:** Elasticsearch
**Created:** [February 2, 2023, 2:03pm UTC](https://discuss.elastic.co/t/elastic-search-does-not-work-during-operation-causing-problems-in-restarting/324556 "2023-02-02T14:03:47Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![sramana235](https://avatars.discourse-cdn.com/v4/letter/s/bbce88/32.png) [@sramana235](https://discuss.elastic.co/u/sramana235)
#### Post date: [February 2, 2023, 2:03pm UTC](https://discuss.elastic.co/t/elastic-search-does-not-work-during-operation-causing-problems-in-restarting/324556/1 "2023-02-02T14:03:47Z")

</div>

```auto
[2023-02-02T18:22:39,230][INFO][o.e.e.NodeEnvironment] [name] using [1] data paths, mounts [[/data (/dev/data)]], net usable_space [55.9gb], net total_space [98.4gb], types [ext4]
[2023-02-02T18:22:39,231][INFO][o.e.e.NodeEnvironment] [name] heap size [2gb], compressed ordinary object pointers [true]
[2023-02-02T18:22:39,352][INFO][o.e.n.Node] [name] node name [name], node ID [NSSQNNryRkeZsMxOuFd7gQ], cluster name [cluster_name], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2023-02-02T18:22:43,029][INFO][o.e.x.m.p.l.CppLogMessageHandler] [name] [controller/26337] [Main.cc@122] controller (64 bit): Version 7.16.3 (Build b71e71b93f3e4d) Copyright (c) 2021 Elasticsearch BV
[2023-02-02T18:22:43,529][INFO][o.e.x.s.a.Realms] [name] license mode is [trial], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-02-02T18:22:43,536][INFO][o.e.x.s.a.s.FileRolesStore] [name] parsed [0] roles from file [/data/elasticsearch/config/roles.yml]
[2023-02-02T18:22:43,830][ERROR][o.e.b.Bootstrap] [name] Exception
java.io.UncheckedIOException: Failed to create persistent cache writers
    at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createWriters(PersistentCache.java:346) ~[?:?]
    at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.<init>(PersistentCache.java:100) ~[?:?]
    at org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots.createComponents(SearchableSnapshots.java:333) ~[?:?]
    at org.elasticsearch.node.Node.lambda$new$18(Node.java:733) ~[elasticsearch-7.16.3.jar:7.16.3]
    at java.util.stream.ReferencePipeline$7$1.accept(Unknown Source) ~[?:?]
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) ~[?:?]
    at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
    at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
    at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
    at org.elasticsearch.node.Node.<init>(Node.java:747) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) [elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) [elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) [elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) [elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-7.16.3.jar:7.16.3]
    at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) [elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.16.3.jar:7.16.3]
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by another program: /data/elasticsearch/data/nodes/0/snapshot_cache/write.lock
    at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:130) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
    at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
    at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:923) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
    at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createCacheIndexWriter(PersistentCache.java:377) ~[?:?]
    at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createWriters(PersistentCache.java:342) ~[?:?]
    ... 22 more
[2023-02-02T18:22:43,835][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [name] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.io.UncheckedIOException: Failed to create persistent cache writers
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.3.jar:7.16.3]
    at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.3.jar:7.16.3]
Caused by: java.io.UncheckedIOException: Failed to create persistent cache writers
    at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createWriters(PersistentCache.java:346) ~[?:?]
    at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.<init>(PersistentCache.java:100) ~[?:?]
    at org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots.createComponents(SearchableSnapshots.java:333) ~[?:?]
    at org.elasticsearch.node.Node.lambda$new$18(Node.java:733) ~[elasticsearch-7.16.3.jar:7.16.3]
    at java.util.stream.ReferencePipeline$7$1.accept(Unknown Source) ~[?:?]
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) ~[?:?]
    at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
    at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
    at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
    at org.elasticsearch.node.Node.<init>(Node.java:747) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.3.jar:7.16.3]
    ... 6 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by another program: /data/elasticsearch/data/nodes/0/snapshot_cache/write.lock
    at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:130) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
    at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
    at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:923) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
    at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createCacheIndexWriter(PersistentCache.java:377) ~[?:?]
    at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.createWriters(PersistentCache.java:342) ~[?:?]
    at org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache.<init>(PersistentCache.java:100) ~[?:?]
    at org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots.createComponents(SearchableSnapshots.java:333) ~[?:?]
    at org.elasticsearch.node.Node.lambda$new$18(Node.java:733) ~[elasticsearch-7.16.3.jar:7.16.3]
    at java.util.stream.ReferencePipeline$7$1.accept(Unknown Source) ~[?:?]
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) ~[?:?]
    at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
    at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
    at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
    at org.elasticsearch.node.Node.<init>(Node.java:747) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.3.jar:7.16.3]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.3.jar:7.16.3]
    ... 6 more
[2023-02-02T18:22:43,869][INFO][o.e.x.m.p.NativeController] [name] Native controller process has stopped - no new native processes can be started

```

The first log I found is as above. I tried everything I found on Google to solve this problem, but it failed.

```auto
[2023-02-02T21:00:36,072][INFO][o.e.e.NodeEnvironment] [name] using [1] data paths, mounts [[/data (/dev/data)]], net usable_space [55.6gb], net total_space [98.4gb], types [ext4]
[2023-02-02T21:00:36,072][INFO][o.e.e.NodeEnvironment] [name] heap size [2gb], compressed ordinary object pointers [true]
[2023-02-02T21:00:36,116][INFO][o.e.n.Node] [name] node name [name], node ID [ljP7-iEISP2bDFseryxhhA], cluster name [cluster_name], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2023-02-02T21:00:39,781][INFO][o.e.x.m.p.l.CppLogMessageHandler] [name] [controller/30735] [Main.cc@122] controller (64 bit): Version 7.16.3 (Build b71e71b93f3e4d) Copyright (c) 2021 Elasticsearch BV
[2023-02-02T21:00:40,136][INFO][o.e.x.s.a.Realms] [name] license mode is [trial], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-02-02T21:00:40,145][INFO][o.e.x.s.a.s.FileRolesStore] [name] parsed [0] roles from file [/data/elasticsearch/config/roles.yml]
[2023-02-02T21:00:40,690][INFO][o.e.i.g.ConfigDatabases] [name] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/data/elasticsearch/config/ingest-geoip] for changes
[2023-02-02T21:00:40,691][INFO][o.e.i.g.DatabaseNodeService] [name] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-16478923630485953060/geoip-databases/ljP7-iEISP2bDFseryxhhA]
[2023-02-02T21:00:41,185][INFO][o.e.t.NettyAllocator] [name] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2023-02-02T21:00:41,241][INFO][o.e.d.DiscoveryModule] [name] using discovery type [zen] and seed hosts providers [settings]
[2023-02-02T21:00:41,647][INFO][o.e.g.DanglingIndicesState] [name] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-02-02T21:00:42,081][INFO][o.e.n.Node] [name] initialized
[2023-02-02T21:00:42,082][INFO][o.e.n.Node] [name] starting ...
[2023-02-02T21:00:42,113][INFO][o.e.x.s.c.f.PersistentCache] [name] persistent cache index loaded
[2023-02-02T21:00:42,115][INFO][o.e.x.d.l.DeprecationIndexingComponent] [name] deprecation component started
[2023-02-02T21:00:42,193][INFO][o.e.t.TransportService] [name] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2023-02-02T21:00:42,454][WARN][o.e.b.BootstrapChecks] [name] memory locking requested for elasticsearch process but memory is not locked
[2023-02-02T21:00:42,456][INFO][o.e.c.c.Coordinator] [name] cluster UUID [eB0SDrwlSP-905wyIowBFQ]
[2023-02-02T21:00:42,578][INFO][o.e.c.s.MasterService] [name] elected-as-master ([1] nodes joined)[{name}{ljP7-iEISP2bDFseryxhhA}{uuqbJScFQsWlGI-8wfv6qA}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 3, version: 63, delta: master node changed {previous [], current [{name}{ljP7-iEISP2bDFseryxhhA}{uuqbJScFQsWlGI-8wfv6qA}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}
[2023-02-02T21:00:42,650][INFO][o.e.c.s.ClusterApplierService] [name] master node changed {previous [], current [{name}{ljP7-iEISP2bDFseryxhhA}{uuqbJScFQsWlGI-8wfv6qA}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}, term: 3, version: 63, reason: Publication{term=3, version=63}
[2023-02-02T21:00:42,698][INFO][o.e.h.AbstractHttpServerTransport] [name] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2023-02-02T21:00:42,699][INFO][o.e.n.Node] [name] started
[2023-02-02T21:00:42,971][INFO][o.e.l.LicenseService] [name] license [9903a757-0105-4659-8056-6974082ed965] mode [basic] - valid
[2023-02-02T21:00:42,972][INFO][o.e.x.s.a.Realms] [name] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-02-02T21:00:42,973][INFO][o.e.x.s.s.SecurityStatusChangeListener] [name] Active license is now [BASIC]; Security is enabled
[2023-02-02T21:00:42,976][INFO][o.e.g.GatewayService] [name] recovered [2] indices into cluster_state
[2023-02-02T21:00:43,346][INFO][o.e.c.r.a.AllocationService] [name] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.ds-.logs-deprecation.elasticsearch-default-2023.02.02-000001][0], [.ds-ilm-history-5-2023.02.02-000001][0]]]).

```

Afterwards, I deleted and reinstalled the Elasticsearch, and the logs I found were as follows.  
It also tried everything, but it didn't succeed.

Thereafter, two logs are generated alternately.  
I wanted to replace the hard disk, but I couldn't because I couldn't do that.

Please let me know if the hard disk is a problem or if there is another solution.

---

<div class="post-metadata">

### Author: ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)
#### Post date: [February 2, 2023, 3:00pm UTC](https://discuss.elastic.co/t/elastic-search-does-not-work-during-operation-causing-problems-in-restarting/324556/2 "2023-02-02T15:00:10Z")

</div>

I think this is the problem

> memory locking requested for elasticsearch process but memory is not locked

Is [this option](https://www.elastic.co/guide/en/elasticsearch/reference/current/_memory_lock_check.html) turned on? You'll have to see why the memory lock isn't working in your environment.

---

<div class="post-metadata">

### Author: ![sramana235](https://avatars.discourse-cdn.com/v4/letter/s/bbce88/32.png) [@sramana235](https://discuss.elastic.co/u/sramana235)
#### Post date: [February 3, 2023, 12:47am UTC](https://discuss.elastic.co/t/elastic-search-does-not-work-during-operation-causing-problems-in-restarting/324556/3 "2023-02-03T00:47:03Z")

</div>

```auto
cluster.name: cluster_name
node.name: name
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/logs
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["0.0.0.0"]
cluster.initial_master_nodes: ["name"]
ingest.geoip.downloader.enabled: false

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

```

My settings are as follows.  
I checked if there is a problem with the license, but it is working as basic.

And I have one more piece of equipment of the same spec and it's working well there.

---

<div class="post-metadata">

### Author: ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)
#### Post date: [February 3, 2023, 7:22am UTC](https://discuss.elastic.co/t/elastic-search-does-not-work-during-operation-causing-problems-in-restarting/324556/4 "2023-02-03T07:22:33Z")

</div>

From the second logs, it looks like ES was up and running and status is green, what is the issue there (after re-installation) ?

> Lock held by another program: /data/elasticsearch/data/nodes/0/snapshot\_cache/write.lock

From the first log, it looks like there was some snapshot in progress which was holding the lock, have you tried deleting that lock file post restarting ES ?

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 3, 2023, 7:23am UTC](https://discuss.elastic.co/t/elastic-search-does-not-work-during-operation-causing-problems-in-restarting/324556/5 "2023-03-03T07:23:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
