ElasticSearch cloud-aws plugin with riak s3: snapshot is done, failed to finalize snapshot

I’m trying to create Elasticsearch snapshot into riak s3 repository using cloud-aws (Elasticsearch plugin).

I’m using:

Red Hat Enterprise Linux Server release 7.2 (Maipo)

java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

ElasticSearch 2.3.2 (lucene_version 5.5.0)

cloud-aws-2.3.2

For Elasticsearch I configured s3 in ES config file (elasticsearch.yml) .

cloud:
    aws:
            access_key: ********************
            secret_key: ****************************************
            endpoint: *******************
            protocol: http
            s3:
                    signer: S3SignerType 

When I register s3 as repository

curl -XPUT 'http://localhost:9200/_snapshot/s3_repo?verify=false' -d 'type": "s3", "settings": {"access_key": "************************", "secret_key": "******************************", "bucket": "es-bucket", "endpoint": "*******************", "signer": "AWS3SignerType"}}'

there is only one warning:

java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer; 

but register seems to be successful. When I list all my registred repositories there is my s3.
The problem is with creating snapshot – it’s always failed.

My snapshot:

{
 "snapshots": [ {
    "snapshot":"snapshot_2",
    "version_id":2030299,
    "version":"2.3.2",
    "indices":["test"],
    "state":"PARTIAL",
    "start_time":"2016-05-10T06:59:40.975Z",
    "start_time_in_millis":1462863580975,
    "duration_in_millis":124,
    "failures": [ { 
      "index":"test",
      "shard_id":4,
      "reason":"IndexShardSnapshotFailedException[java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;]; nested: NoSuchMethodError[java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;]; ",
      "node_id":"M7xutdTSTlqTGUelqNglrA",
      "status":"INTERNAL_SERVER_ERROR"
    },
    {
      "index":"test",
      "shard_id":2,
      "reason":"IndexShardSnapshotFailedException[java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;]; nested: NoSuchMethodError[java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;]; ",
      "node_id":"M7xutdTSTlqTGUelqNglrA",
      "status":"INTERNAL_SERVER_ERROR"
    },
    (...)
    } ],
    "shards" : {
      "total":5,
      "failed":5,
      "successful":0
    }
  } ]

I tested this plugin with Amazon s3 and it works without issues so I assume that is problem with riak s3. Am I right?

I've tested for both java - oracle and open.
s3cmd and s3fs works propertly for this s3.

Here are Elasticsearch logs :

    [2016-05-10 12:38:05,098][INFO ][snapshots                ] [Morbius]  snapshot [s3_backup:snapshot_1] is done
    [2016-05-10 12:38:05,159][WARN ][snapshots                ] [Morbius] [s3_backup:snapshot_1] failed to finalize snapshot
    java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
        at org.apache.http.client.utils.URLEncodedUtils.urlDecode(URLEncodedUtils.java:521)
        at org.apache.http.client.utils.URLEncodedUtils.decodeFormFields(URLEncodedUtils.java:550)
        at org.apache.http.client.utils.URLEncodedUtils.parse(URLEncodedUtils.java:252)
        at org.apache.http.client.utils.URLEncodedUtils.parse(URLEncodedUtils.java:222)
        at org.apache.http.client.utils.URIBuilder.parseQuery(URIBuilder.java:95)
        at org.apache.http.client.utils.URIBuilder.digestURI(URIBuilder.java:165)
        at org.apache.http.client.utils.URIBuilder.<init>(URIBuilder.java:90)
        at org.apache.http.client.utils.URIUtils.rewriteURI(URIUtils.java:138)
        at org.apache.http.impl.client.DefaultRequestDirector.rewriteRequestURI(DefaultRequestDirector.java:354)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:477)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:864)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
        at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:769)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:506)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:318)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3548)
        at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:647)
        at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:632)
        at org.elasticsearch.cloud.aws.blobstore.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:117)
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshots(BlobStoreRepository.java:398)
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.finalizeSnapshot(BlobStoreRepository.java:376)
        at org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:797)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

Please format your posts.

Is it a fresh install and unmodified version of elasticsearch?

Asking this because it sounds like you have 2 different versions on a dependency.

Are you running elasticsearch embedded? I don't think so but I prefer asking.

Sorry, I had some problems with formatting.

I have two instances: first connected to RiakCS S3 and second connected to Amazon S3 (created just to test how cloud-aws plugin works). Both with fresh install and one index named test.

I'm not sure what is running elasticsearch embedded.
Did you mean https://github.com/fabn/elasticsearch-embedded ?

I start elasticsearch with init.d:
/etc/init.d/elasticsearch start

So it's a standalone instance. Great.

What are the plugins you are using? What does this give? GET _cat/plugins?v

I'm using only cloud-aws 2.3.2

But can you run this please: GET _cat/plugins?v?

And paste the full result here? (formatted with </> icon)

My instance has gone and when I installed everything once again it working without any exception.
It was probably problem with riak s3 and now it's solved.

Thank you for your help and quick response.

P.S.
My result was:
Frankie and Victoria cloud-aws 2.3.2 j