Unable to create elasticserach backup on glusterfs

[Snehit_Gajjar]

I have 3-node elasticsearch cluster. I had issue where snapshots were not created. At that time I did not have shared filesystem across all three nodes. So I went ahead and used glusterfs to create shared filesystem. Now when I try to create a snapshot. I get the same error.

Input:

$ curl -X PUT "localhost:9200/_snapshot/my_backup" -H 'Content-Type: application/json' -d'
>{
>   "type": "fs",
>   "settings": {
>     "location": "my_backup_location"
>   }
> }
> '

Output:

{  
  "error":{  
    "root_cause":[  
       {  
          "type":"repository_verification_exception",
          "reason":"[my_backup] [[IMJUAVQaSt6oeHybvCs_nw, 'RemoteTransportException[[IMJUAVQ][104.170.209.16:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] a file written by master to the store [/var/es_backup/backup/my_backup_location] cannot be accessed on the node [{IMJUAVQ}{IMJUAVQaSt6oeHybvCs_nw}{pdYRCwdpRo66iVc4RHF98g}{104.170.209.16}{104.170.209.16:9300}]. This might indicate that the store [/var/es_backup/backup/my_backup_location] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];'], [3PUdIddHRsCeT2n-dl0ikw, 'RemoteTransportException[[3PUdIdd][104.170.209.17:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] a file written by master to the store [/var/es_backup/backup/my_backup_location] cannot be accessed on the node [{3PUdIdd}{3PUdIddHRsCeT2n-dl0ikw}{sqUGInz2Q2uDp4Xa5tRYGQ}{104.170.209.17}{104.170.209.17:9300}]. This might indicate that the store [/var/es_backup/backup/my_backup_location] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];']]"
       }
    ],
    "type":"repository_verification_exception",
    "reason":"[my_backup] [[IMJUAVQaSt6oeHybvCs_nw, 'RemoteTransportException[[IMJUAVQ][104.170.209.16:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] a file written by master to the store [/var/es_backup/backup/my_backup_location] cannot be accessed on the node [{IMJUAVQ}{IMJUAVQaSt6oeHybvCs_nw}{pdYRCwdpRo66iVc4RHF98g}{104.170.209.16}{104.170.209.16:9300}]. This might indicate that the store [/var/es_backup/backup/my_backup_location] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];'], [3PUdIddHRsCeT2n-dl0ikw, 'RemoteTransportException[[3PUdIdd][104.170.209.17:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] a file written by master to the store [/var/es_backup/backup/my_backup_location] cannot be accessed on the node [{3PUdIdd}{3PUdIddHRsCeT2n-dl0ikw}{sqUGInz2Q2uDp4Xa5tRYGQ}{104.170.209.17}{104.170.209.17:9300}]. This might indicate that the store [/var/es_backup/backup/my_backup_location] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];']]" 
   }, 
  "status":500  
}

It would be nice to get some opinion on use of glusterfs.