Translog Corruption- AccessDeniedException -in ElasticSearch

Elastic Stack Elasticsearch
1

Hi,
I came across few instances where the shards went into "Unassigned" state and failed to recover.
The logs indicated a possible translog corruption with the following exception-

IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];

Any help to reason with why this might be happening?

Thanks in advance!
~Divya Malini

2

Can you post the entire error?

3

Pasted below is the entire error from the logs -

[2017-09-18 11:54:38,003][WARN ][indices.cluster          ] [DEV-ENVIRON] [[myindex1][5]] marking and sending shard failed due to [failed recovery]
[myindex1][[myindex1][5]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
              at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
              at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
Caused by: [myindex1][[myindex1][5]] EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:152)
              at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
              at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)
              at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)
              at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)
              at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
              ... 5 more
Caused by: java.nio.file.AccessDeniedException: G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog
              at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
              at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:504)
              at java.nio.file.Files.createDirectory(Files.java:674)
              at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
              at java.nio.file.Files.createDirectories(Files.java:767)
              at org.elasticsearch.index.translog.Translog.<init>(Translog.java:200)
              at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:205)
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:148)
              ... 11 more
[2017-09-18 11:54:38,003][WARN ][cluster.action.shard     ] [DEV-ENVIRON] [myindex1][5] received shard failed for target shard [[myindex1][5], node[xioUaZjeTeOmOOoxfE819Q], [P], v[1], s[INITIALIZING], a[id=5IxfHoQ-QUCf0TlOpHbF-Q], unassigned_info[[reason=INDEX_CREATED], at[2017-09-18T09:54:37.831Z]]], indexUUID [nXf_w5PLTtKPCnZG2Lpqsw], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog]; ]
[myindex1][[myindex1][5]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
              at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
              at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
Caused by: [myindex1][[myindex1][5]] EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:152)
              at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
              at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)
              at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)
              at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)
              at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
              ... 5 more
Caused by: java.nio.file.AccessDeniedException: G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog
              at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
              at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:504)
              at java.nio.file.Files.createDirectory(Files.java:674)
              at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
              at java.nio.file.Files.createDirectories(Files.java:767)
              at org.elasticsearch.index.translog.Translog.<init>(Translog.java:200)
              at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:205)
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:148)
              ... 11 more
4

Is the G: drive on that system a remote mount, or a local disk.

5

G: drive is present in the system.

6

It's a local disk?

7

yes, it's a local disk

8

Hi Mark,
Any reason on why could this corruption have happened with the AccessDeniedException, could you help?

9

I was going through the logs again. This corruption occurred while creating a new index, Also before the creation of index the cluster state was green as seen in the logs. So what could be the reason for corruption of an index that is newly getting created? Any pointers here would be helpful.

[2017-09-18 11:52:11,059][INFO ][cluster.routing.allocation] [DEV-ENVIRON] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[oldindex][3]] ...]).
> [2017-09-18 11:53:49,053][INFO ][cluster.metadata ] [DEV-ENVIRON] [myindex1] creating index, cause [api], templates [], shards [12]/[0], mappings [workContract]
2017-09-18 11:54:38,003][WARN ][indices.cluster ] [DEV-ENVIRON] [[myindex1][5]] marking and sending shard failed due to [failed recovery]
[myindex1][[myindex1][5]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];

10

That points to Elasticsearch not having the correct permissions to write the file?

11

What permission it was denied is something that isn't clear. The account by default has read/write/execute..

And this one could be because of ?

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.