Getting Repository verification exception while creating Snapshot

[manjukdvg (Manjunath)]

I am trying to create snapshot for S3 repository . But it's throwing exception .
Elasticsearch version : 2.3.5 using plugin cloud-aws . Plugin is installed across all the nodes.

I have created AWS user with permission as below .

{
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::TESTBUCKET"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::TESTBUCKET/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

and trying to execute the following command .
PUT /_snapshot/dev_snapshot1
{
"type": "s3",
"settings": {
"bucket": "TESTBUCKET",
"base_path": "backups/",
"access_key": "ACCESS_KEY_HERE",
"secret_key": "SECRET_KEY_HERE",
"endpoint": "s3-us-west-2.amazonaws.com"
}
}

I am getting the following exception.

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[dev_snapshot1] [uKBReIYMTs6ALUY4pHVC-A, 'RemoteTransportException[[data_4][IP1:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[dev_snapshot1] missing];'], [tAxayK0uTz-ACFRPdovzXQ, 'RemoteTransportException[[data_5][IP2.207:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[dev_snapshot1] missing];'], [cVeA7vYRTBqurWdDcQKeag, 'RemoteTransportException[[data_3][IP3:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[dev_snapshot1] missing];']]"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[dev_snapshot1] [uKBReIYMTs6ALUY4pHVC-A, 'RemoteTransportException[[data_4][IP1:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[dev_snapshot1] missing];'], [tAxayK0uTz-ACFRPdovzXQ, 'RemoteTransportException[[data_5][IP2:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[dev_snapshot1] missing];'], [cVeA7vYRTBqurWdDcQKeag, 'RemoteTransportException[[data_3][IP3:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[dev_snapshot1] missing];']]"
  },
  "status": 500
}

What could be wrong here ?

[theuntergeek (Aaron Mildenstein)]

What this is suggesting is that one or more of your local nodes does not have write access to the remote S3 end point, or that the write was slow or interrupted so it did not or could not complete.

The repository verification step occurs at repository creation time. It tells each node to write a file to the remote end point, and then check to ensure the file got written. The message you're getting is that it couldn't find one of those files sent by one or more nodes, for whatever reason.

[manjukdvg (Manjunath)]

Yes . Got it . One of the data node had issue . Now working fine. Thank you.