Elasticsearch running on Docker failing in taking snapshots and restore

620528 · March 30, 2020, 6:17am

Below is the logs taken from Curator while running snapshot. Can someone please help me on this ?


2020-03-30 02:02:52,095 WARNING            elasticsearch       log_request_fail:149  PUT http://localhost:9200/_snapshot/backup/indices-2020.03.30?wait_for_completion=false [status:500 request:0.551s]
2020-03-30 02:02:52,095 DEBUG              elasticsearch       log_request_fail:160  > {"ignore_unavailable":false,"include_global_state":true,"partial":false,"indices":"index-2019.04.30"}
2020-03-30 02:02:52,095 DEBUG              elasticsearch       log_request_fail:165  < {"error":{"root_cause":[{"type":"exception","reason":"failed to create blob container"}],"type":"exception","reason":"failed to create blob container","caused_by":{"type":"access_denied_exception","reason":"/mnt/esbackup/indices/8qdN1CiuQD-sMACmRWbIIA"}},"status":500}
2020-03-30 02:02:52,095 ERROR                curator.cli                    run:213  Failed to complete action: snapshot.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: TransportError(500, 'exception', 'failed to create blob container')

Christian_Dahlqvist · March 30, 2020, 6:19am

What does the Elasticsearch logs contain?

620528 · March 30, 2020, 6:27am

Thanks @Christian_Dahlqvist for the quick response.

It has the very same message saying its not able to access this path given on curator. Access denied only. I am using Elastic 5.6.16 on Docker. I will get you the elasticsearch logs in a while, however with the current info what do you think i am doing wrong here. Its happening only on PROD environment. In lower environment its working perfectly fine.

Its not only happening via Cerebro and also its happening via Cerebro API hit. Access denied exception.

Christian_Dahlqvist · March 30, 2020, 6:40am

I do not know as I have not seen the Elasticsearch logs, which likely will contain a lot more information. It could be that all nodes do not have access to the shared file system or that it has been mounted incorrectly.

620528 · March 30, 2020, 6:48am

@Christian_Dahlqvist As it is a PROD environment, i couldn't get the logs at this moment. I will share the elasticsearch logs by this evening.

620528 · March 30, 2020, 2:15pm

This is the elasticsearch logs we obtained on PROD servers. Can someone please help @Christian_Dahlqvist ?

[2020-03-30T12:01:06,781][WARN ][r.suppressed             ] path: /_snapshot/analytics_backup/_verify, params: {repository=analytics_backup}
org.elasticsearch.transport.RemoteTransportException: [VFPV-A01ESM03][10.99.64.93:9300][cluster:admin/repository/verify]
Caused by: org.elasticsearch.ElasticsearchException: failed to create blob container
	at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:70) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:720) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.repositories.RepositoriesService.verifyRepository(RepositoriesService.java:210) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.action.admin.cluster.repositories.verify.TransportVerifyRepositoryAction.masterOperation(TransportVerifyRepositoryAction.java:70) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.action.admin.cluster.repositories.verify.TransportVerifyRepositoryAction.masterOperation(TransportVerifyRepositoryAction.java:40) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.16.jar:5.6.16]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: java.nio.file.AccessDeniedException: /mnt/bkup/elasticsearchbkup/analytics_backup/tests-Mgn1-6jxSJSaBdJIWeNOgg
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:?]
	at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_212]
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_212]
	at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_212]
	at org.elasticsearch.common.blobstore.fs.FsBlobStore.buildAndCreate(FsBlobStore.java:86) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:68) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:720) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.repositories.RepositoriesService.verifyRepository(RepositoriesService.java:210) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.action.admin.cluster.repositories.verify.TransportVerifyRepositoryAction.masterOperation(TransportVerifyRepositoryAction.java:70) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.action.admin.cluster.repositories.verify.TransportVerifyRepositoryAction.masterOperation(TransportVerifyRepositoryAction.java:40) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.16.jar:5.6.16]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]

620528 · March 31, 2020, 10:58am

Can someone please help us on this issue ?

system · April 28, 2020, 10:58am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.