Translog Corruption- AccessDeniedException -in ElasticSearch


(Divya Malini) #1

Hi,
I came across few instances where the shards went into "Unassigned" state and failed to recover.
The logs indicated a possible translog corruption with the following exception-

IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];

Any help to reason with why this might be happening?

Thanks in advance!
~Divya Malini


(Mark Walkom) #2

Can you post the entire error?


(Divya Malini) #3

Pasted below is the entire error from the logs -

[2017-09-18 11:54:38,003][WARN ][indices.cluster          ] [DEV-ENVIRON] [[myindex1][5]] marking and sending shard failed due to [failed recovery]
[myindex1][[myindex1][5]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
              at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
              at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
Caused by: [myindex1][[myindex1][5]] EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:152)
              at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
              at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)
              at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)
              at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)
              at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
              ... 5 more
Caused by: java.nio.file.AccessDeniedException: G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog
              at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
              at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:504)
              at java.nio.file.Files.createDirectory(Files.java:674)
              at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
              at java.nio.file.Files.createDirectories(Files.java:767)
              at org.elasticsearch.index.translog.Translog.<init>(Translog.java:200)
              at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:205)
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:148)
              ... 11 more
[2017-09-18 11:54:38,003][WARN ][cluster.action.shard     ] [DEV-ENVIRON] [myindex1][5] received shard failed for target shard [[myindex1][5], node[xioUaZjeTeOmOOoxfE819Q], [P], v[1], s[INITIALIZING], a[id=5IxfHoQ-QUCf0TlOpHbF-Q], unassigned_info[[reason=INDEX_CREATED], at[2017-09-18T09:54:37.831Z]]], indexUUID [nXf_w5PLTtKPCnZG2Lpqsw], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog]; ]
[myindex1][[myindex1][5]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
              at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
              at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
Caused by: [myindex1][[myindex1][5]] EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:152)
              at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
              at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)
              at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)
              at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)
              at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)
              at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
              ... 5 more
Caused by: java.nio.file.AccessDeniedException: G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog
              at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
              at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
              at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:504)
              at java.nio.file.Files.createDirectory(Files.java:674)
              at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
              at java.nio.file.Files.createDirectories(Files.java:767)
              at org.elasticsearch.index.translog.Translog.<init>(Translog.java:200)
              at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:205)
              at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:148)
              ... 11 more

(Mark Walkom) #4

Is the G: drive on that system a remote mount, or a local disk.


(Divya Malini) #5

G: drive is present in the system.


(Mark Walkom) #6

It's a local disk?


(Divya Malini) #7

yes, it's a local disk


(Divya Malini) #8

Hi Mark,
Any reason on why could this corruption have happened with the AccessDeniedException, could you help?


(Divya Malini) #9

I was going through the logs again. This corruption occurred while creating a new index, Also before the creation of index the cluster state was green as seen in the logs. So what could be the reason for corruption of an index that is newly getting created? Any pointers here would be helpful.

[2017-09-18 11:52:11,059][INFO ][cluster.routing.allocation] [DEV-ENVIRON] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[oldindex][3]] ...]).
> [2017-09-18 11:53:49,053][INFO ][cluster.metadata ] [DEV-ENVIRON] [myindex1] creating index, cause [api], templates [], shards [12]/[0], mappings [workContract]
2017-09-18 11:54:38,003][WARN ][indices.cluster ] [DEV-ENVIRON] [[myindex1][5]] marking and sending shard failed due to [failed recovery]
[myindex1][[myindex1][5]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: AccessDeniedException[G:\Data\Search\IndexStore\MyComp\nodes\0\indices\myindex1\5\translog];


(Mark Walkom) #10

That points to Elasticsearch not having the correct permissions to write the file?


(Divya Malini) #11

What permission it was denied is something that isn't clear. The account by default has read/write/execute..

And this one could be because of ?


(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.