Elasticsearch Cluster Multi Node Shared File System Repository SMB Issue

Elastic Stack Elasticsearch
1

Hi Team! I am trying to register shared file system repository in elasticsearch 8.6.2. I have windows server where i have created a directory and have shared it using smb and have mounted that directory using NFS CIFS on all elasticsearch nodes but when I register the repository in Kibana. I am receiving the following error. I have tested by manually creating and deleting files on the folder that is mounted and verified that it is working but somehow elastic is unable to verify it. I have tested the same scenario with linux nfs server as well and it is working fine without any issues but the problem is with windows smb. Any help would be appreciated

  • I have installed 'nfs-common cifs-utils '

  • Using below command all three nodes of cluster to setup where 114,120 is elasticsearch uid and gid

  • I have installed nfs-common cifs-utils -y

  • Using below command to setup where 114,120 is elasticsearch uid and gid
    mount -t cifs //IP/lab01-elastic-security-backup /home/lab01-elastic-security-backup/ -o credentials=/root/.nfscredentials,uid=114,gid=120

 {
   "name": "ResponseError",
   "meta": {
     "body": {
       "error": {
         "root_cause": [
           {
             "type": "repository_verification_exception",
             "reason": "[Lab01] cannot delete test data at "
           }
         ],
         "type": "repository_verification_exception",
         "reason": "[Lab01] cannot delete test data at ",
         "caused_by": {
           "type": "directory_not_empty_exception",
           "reason": "/home/lab01-elastic-security-backup/tests-PiBrldzOS3uFAW8KflFPTg"
         }
       },
       "status": 500
     },
     "statusCode": 500,
     "headers": {
       "x-opaque-id": "4775fbd1-1a64-4b1c-88e4-542f381afeb2;kibana:application:management:",
       "x-elastic-product": "Elasticsearch",
       "content-type": "application/json;charset=utf-8",
       "content-length": "350"
     },
     "meta": {
       "context": null,
       "request": {
         "params": {
           "method": "POST",
           "path": "/_snapshot/Lab01/_verify",
           "querystring": "",
           "headers": {
             "user-agent": "Kibana/8.6.2",
             "x-elastic-product-origin": "kibana",
             "authorization": "Basic c29jOlJEQ0BmM3RlY2gxMjM=",
             "x-opaque-id": "4775fbd1-1a64-4b1c-88e4-542f381afeb2;kibana:application:management:",
             "x-elastic-client-meta": "es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1",
             "accept": "application/vnd.elasticsearch+json; compatible-with=8,text/plain"
           }
         },
         "options": {
           "opaqueId": "4775fbd1-1a64-4b1c-88e4-542f381afeb2;kibana:application:management:",
           "headers": {
             "x-elastic-product-origin": "kibana",
             "user-agent": "Kibana/8.6.2",
             "authorization": "Basic c29jOlJEQ0BmM3RlY2gxMjM=",
             "x-opaque-id": "4775fbd1-1a64-4b1c-88e4-542f381afeb2",
             "x-elastic-client-meta": "es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1"
           }
         },
         "id": 1
       },
       "name": "elasticsearch-js",
       "connection": {
         "url": "https://www.lab01-home.tk:9200/",
         "id": "https://www.lab01-home.tk:9200/",
         "headers": {},
         "status": "alive"
       },
       "attempts": 0,
       "aborted": false
     },
     "warnings": null
   }
 }
2

Elasticsearch creates a directory, adds some files, lists the directory contents, deletes all the files it found, and finally deletes the directory. The last step is failing with the error above which indicates that the directory is not empty.

I'm not sure there's anything we can do on this forum to help further, because the problem lies in your storage system and not Elasticsearch.

3

The directory is completely empty and freshly setup for snapshots with the read and write privileges

4

I don't doubt it, but that's not related to the problem I described above.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.