Translog Corruption- AccessDeniedException -in ElasticSearch

dimalini · September 24, 2017, 6:13pm

Hi,
I came across few instances where the shards went into "Unassigned" state and failed to recover.
The logs indicated a possible translog corruption with the following exception-

IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];

Any help to reason with why this might be happening?

Thanks in advance!
~Divya Malini

warkolm · September 26, 2017, 1:48am

Can you post the entire error?

dimalini · September 26, 2017, 4:07am

Pasted below is the entire error from the logs -

[2017-09-18 11:54:38,003][WARN ][indices.cluster          ] [DEV-ENVIRON] [[myindex1][5]] marking and sending shard failed due to [failed recovery]
[myindex1][[myindex1][5]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
              at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
              at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
Caused by: [myindex1][[myindex1][5]] EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:152)
              at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
              at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)
              at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)
              at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)
              at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
              ... 5 more
Caused by: java.nio.file.AccessDeniedException: G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog
              at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
              at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:504)
              at java.nio.file.Files.createDirectory(Files.java:674)
              at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
              at java.nio.file.Files.createDirectories(Files.java:767)
              at org.elasticsearch.index.translog.Translog.<init>(Translog.java:200)
              at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:205)
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:148)
              ... 11 more
[2017-09-18 11:54:38,003][WARN ][cluster.action.shard     ] [DEV-ENVIRON] [myindex1][5] received shard failed for target shard [[myindex1][5], node[xioUaZjeTeOmOOoxfE819Q], [P], v[1], s[INITIALIZING], a[id=5IxfHoQ-QUCf0TlOpHbF-Q], unassigned_info[[reason=INDEX_CREATED], at[2017-09-18T09:54:37.831Z]]], indexUUID [nXf_w5PLTtKPCnZG2Lpqsw], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog]; ]
[myindex1][[myindex1][5]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
              at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
              at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
Caused by: [myindex1][[myindex1][5]] EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:152)
              at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
              at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)
              at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)
              at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)
              at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
              ... 5 more
Caused by: java.nio.file.AccessDeniedException: G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog
              at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
              at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:504)
              at java.nio.file.Files.createDirectory(Files.java:674)
              at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
              at java.nio.file.Files.createDirectories(Files.java:767)
              at org.elasticsearch.index.translog.Translog.<init>(Translog.java:200)
              at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:205)
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:148)
              ... 11 more

warkolm · September 26, 2017, 4:41am

Is the G: drive on that system a remote mount, or a local disk.

dimalini · September 26, 2017, 4:46am

G: drive is present in the system.

warkolm · September 26, 2017, 4:50am

It's a local disk?

dimalini · September 26, 2017, 4:52am

yes, it's a local disk

dimalini · October 4, 2017, 7:15am

Hi Mark,
Any reason on why could this corruption have happened with the AccessDeniedException, could you help?

dimalini · October 10, 2017, 9:34am

I was going through the logs again. This corruption occurred while creating a new index, Also before the creation of index the cluster state was green as seen in the logs. So what could be the reason for corruption of an index that is newly getting created? Any pointers here would be helpful.

[2017-09-18 11:52:11,059][INFO ][cluster.routing.allocation] [DEV-ENVIRON] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[oldindex][3]] ...]).
> [2017-09-18 11:53:49,053][INFO ][cluster.metadata ] [DEV-ENVIRON] [myindex1] creating index, cause [api], templates [], shards [12]/[0], mappings [workContract]
2017-09-18 11:54:38,003][WARN ][indices.cluster ] [DEV-ENVIRON] [[myindex1][5]] marking and sending shard failed due to [failed recovery]
[myindex1][[myindex1][5]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];

warkolm · October 10, 2017, 9:44pm

That points to Elasticsearch not having the correct permissions to write the file?

dimalini · October 11, 2017, 3:05am

What permission it was denied is something that isn't clear. The account by default has read/write/execute..

And this one could be because of ?

system · November 8, 2017, 3:06am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elastic Translog corrupted error (Unassigned shards) Elasticsearch	2	169	November 15, 2023
Failed to recover from translog Elasticsearch	3	2043	July 5, 2017
Corrupt translog entry error in index recovery Elasticsearch	1	734	July 6, 2017
How to recover indices from missing translog file Elasticsearch	2	3038	July 5, 2017
Translog files corrupted, cluster failing to recover Elasticsearch	2	1687	July 5, 2017

Translog Corruption- AccessDeniedException -in ElasticSearch

Related topics