# Translog Corruption- AccessDeniedException -in ElasticSearch

**URL:** <https://discuss.elastic.co/t/translog-corruption-accessdeniedexception-in-elasticsearch/101608>\
**Category:** Elasticsearch\
**Created:** [September 24, 2017, 6:13pm UTC](https://discuss.elastic.co/t/translog-corruption-accessdeniedexception-in-elasticsearch/101608 "2017-09-24T18:13:15Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![dimalini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dimalini/32/22448_2.png) [@dimalini](https://discuss.elastic.co/u/dimalini)\
**Post date:** [September 24, 2017, 6:13pm UTC](https://discuss.elastic.co/t/translog-corruption-accessdeniedexception-in-elasticsearch/101608/1 "2017-09-24T18:13:15Z")

</div>

Hi,  
I came across few instances where the shards went into "Unassigned" state and failed to recover.  
The logs indicated a possible translog corruption with the following exception-

> IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];

Any help to reason with why this might be happening?

Thanks in advance!  
~Divya Malini

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 26, 2017, 1:48am UTC](https://discuss.elastic.co/t/translog-corruption-accessdeniedexception-in-elasticsearch/101608/2 "2017-09-26T01:48:11Z")

</div>

Can you post the entire error?

---

<div class="post-metadata">

**Author:** ![dimalini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dimalini/32/22448_2.png) [@dimalini](https://discuss.elastic.co/u/dimalini)\
**Post date:** [September 26, 2017, 4:07am UTC](https://discuss.elastic.co/t/translog-corruption-accessdeniedexception-in-elasticsearch/101608/3 "2017-09-26T04:07:21Z")

</div>

Pasted below is the entire error from the logs -

```auto
[2017-09-18 11:54:38,003][WARN][indices.cluster] [DEV-ENVIRON] [[myindex1][5]] marking and sending shard failed due to [failed recovery]
[myindex1][[myindex1][5]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
              at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
              at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
Caused by: [myindex1][[myindex1][5]] EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:152)
              at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
              at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)
              at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)
              at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)
              at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
              ... 5 more
Caused by: java.nio.file.AccessDeniedException: G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog
              at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
              at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:504)
              at java.nio.file.Files.createDirectory(Files.java:674)
              at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
              at java.nio.file.Files.createDirectories(Files.java:767)
              at org.elasticsearch.index.translog.Translog.<init>(Translog.java:200)
              at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:205)
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:148)
              ... 11 more
[2017-09-18 11:54:38,003][WARN][cluster.action.shard] [DEV-ENVIRON] [myindex1][5] received shard failed for target shard [[myindex1][5], node[xioUaZjeTeOmOOoxfE819Q], [P], v[1], s[INITIALIZING], a[id=5IxfHoQ-QUCf0TlOpHbF-Q], unassigned_info[[reason=INDEX_CREATED], at[2017-09-18T09:54:37.831Z]]], indexUUID [nXf_w5PLTtKPCnZG2Lpqsw], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog]; ]
[myindex1][[myindex1][5]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
              at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
              at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
Caused by: [myindex1][[myindex1][5]] EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:152)
              at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
              at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)
              at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)
              at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)
              at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
              ... 5 more
Caused by: java.nio.file.AccessDeniedException: G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog
              at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
              at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:504)
              at java.nio.file.Files.createDirectory(Files.java:674)
              at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
              at java.nio.file.Files.createDirectories(Files.java:767)
              at org.elasticsearch.index.translog.Translog.<init>(Translog.java:200)
              at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:205)
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:148)
              ... 11 more

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 26, 2017, 4:41am UTC](https://discuss.elastic.co/t/translog-corruption-accessdeniedexception-in-elasticsearch/101608/4 "2017-09-26T04:41:56Z")

</div>

Is the G: drive on that system a remote mount, or a local disk.

---

<div class="post-metadata">

**Author:** ![dimalini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dimalini/32/22448_2.png) [@dimalini](https://discuss.elastic.co/u/dimalini)\
**Post date:** [September 26, 2017, 4:46am UTC](https://discuss.elastic.co/t/translog-corruption-accessdeniedexception-in-elasticsearch/101608/5 "2017-09-26T04:46:33Z")

</div>

G: drive is present in the system.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 26, 2017, 4:50am UTC](https://discuss.elastic.co/t/translog-corruption-accessdeniedexception-in-elasticsearch/101608/6 "2017-09-26T04:50:29Z")

</div>

It's a local disk?

---

<div class="post-metadata">

**Author:** ![dimalini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dimalini/32/22448_2.png) [@dimalini](https://discuss.elastic.co/u/dimalini)\
**Post date:** [September 26, 2017, 4:52am UTC](https://discuss.elastic.co/t/translog-corruption-accessdeniedexception-in-elasticsearch/101608/7 "2017-09-26T04:52:07Z")

</div>

yes, it's a local disk

---

<div class="post-metadata">

**Author:** ![dimalini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dimalini/32/22448_2.png) [@dimalini](https://discuss.elastic.co/u/dimalini)\
**Post date:** [October 4, 2017, 7:15am UTC](https://discuss.elastic.co/t/translog-corruption-accessdeniedexception-in-elasticsearch/101608/8 "2017-10-04T07:15:38Z")

</div>

Hi Mark,  
Any reason on why could this corruption have happened with the AccessDeniedException, could you help?

---

<div class="post-metadata">

**Author:** ![dimalini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dimalini/32/22448_2.png) [@dimalini](https://discuss.elastic.co/u/dimalini)\
**Post date:** [October 10, 2017, 9:34am UTC](https://discuss.elastic.co/t/translog-corruption-accessdeniedexception-in-elasticsearch/101608/9 "2017-10-10T09:34:38Z")

</div>

I was going through the logs again. This corruption occurred while creating a new index, Also before the creation of index the cluster state was green as seen in the logs. So what could be the reason for corruption of an index that is newly getting created? Any pointers here would be helpful.

> **[2017-09-18 11:52:11,059][INFO][cluster.routing.allocation] [DEV-ENVIRON] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[oldindex][3]] ...]).**  
> **\> [2017-09-18 11:53:49,053][INFO][cluster.metadata] [DEV-ENVIRON] [myindex1] creating index, cause [api], templates [], shards [12]/[0], mappings [workContract]**  
> 2017-09-18 11:54:38,003][WARN][indices.cluster] [DEV-ENVIRON] [[myindex1][5]] marking and sending shard failed due to [failed recovery]  
> [myindex1][[myindex1][5]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 10, 2017, 9:44pm UTC](https://discuss.elastic.co/t/translog-corruption-accessdeniedexception-in-elasticsearch/101608/10 "2017-10-10T21:44:09Z")

</div>

> [@dimalini](#):
>
> AccessDeniedException

That points to Elasticsearch not having the correct permissions to write the file?

---

<div class="post-metadata">

**Author:** ![dimalini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dimalini/32/22448_2.png) [@dimalini](https://discuss.elastic.co/u/dimalini)\
**Post date:** [October 11, 2017, 3:05am UTC](https://discuss.elastic.co/t/translog-corruption-accessdeniedexception-in-elasticsearch/101608/11 "2017-10-11T03:05:44Z")

</div>

What permission it was denied is something that isn't clear. The account by default has read/write/execute..

> [@dimalini](#):
>
> This corruption occurred while creating a new index, Also before the creation of index the cluster state was green as seen in the logs. So what could be the reason for corruption of an index that is newly getting created?

And this one could be because of ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 8, 2017, 3:06am UTC](https://discuss.elastic.co/t/translog-corruption-accessdeniedexception-in-elasticsearch/101608/12 "2017-11-08T03:06:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.