Integration tests involving repository s3 and docker

Hi,
I'm struggling to create an integration test for our code orchestrating snapshots to S3.
I've following setup:
Docker compose of external services:

services:
  s3:
    build: s3/
    ports:
      - "9495:9495"
elasticsearch:
    build: elasticsearch/
    ulimits:
      nofile:
        soft: 65536
        hard: 65536
    ports:
      - "9000:9200"

s3 dockerfile (Should I use different s3 mock?):

FROM node:alpine
RUN npm install -g s3rver
EXPOSE 9495
CMD [ "s3rver", "--hostname", "0.0.0.0", "--port", "9495", "--directory", "/tmp" ]

elasticsearch dockerfile (I've tried various versions of this setup):

FROM docker.elastic.co/elasticsearch/elasticsearch:6.0.1
RUN bin/elasticsearch-plugin install repository-s3
RUN echo 123 | bin/elasticsearch-keystore add --stdin --force s3.client.default.secret_key
RUN echo abc | bin/elasticsearch-keystore add --stdin --force s3.client.default.access_key
RUN echo "s3.client.default.endpoint: s3:9495" >> config/elasticsearch.yml
RUN echo "s3.client.default.protocol: http" >> config/elasticsearch.yml
RUN echo "s3.client.default.proxy.host: s3" >> config/elasticsearch.yml
RUN echo "s3.client.default.proxy.port: 9495" >> config/elasticsearch.yml

Bucket creation:

aws s3api create-bucket \
    --acl public-read-write \
    --bucket snapshot \
    --endpoint-url http://localhost:9495

When I try to register my mock bucket:

PUT _snapshot/my_s3_repository
    {
     "type": "s3",
     "settings": {
         "bucket": "snapshot",
         "canned_acl": "public-read-write"
     }
    }

I get following answer (bucket snapshot DOES exist):

{
  "error": {
    "root_cause": [
      {
        "type": "repository_exception",
        "reason": "[my_s3_repository] failed to create repository"
      }
    ],
    "type": "repository_exception",
    "reason": "[my_s3_repository] failed to create repository",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "The bucket [snapshot] does not exist. Please create it before  creating an s3 snapshot repository backed by it."
    }
  },
  "status": 500
}

With the configuration without proxy I get following answer (I'm able to ping s3 host from elasticsearch container):

{
  "error": {
    "root_cause": [
      {
        "type": "repository_exception",
        "reason": "[my_s3_repository] failed to create repository"
      }
    ],
    "type": "repository_exception",
    "reason": "[my_s3_repository] failed to create repository",
    "caused_by": {
      "type": "sdk_client_exception",
      "reason": "Unable to execute HTTP request: snapshot.s3",
      "caused_by": {
        "type": "unknown_host_exception",
        "reason": "snapshot.s3"
      }
    }
  },
  "status": 500
}

Has anyone done anything similar? Does ElasticSearch use integration test setup of their own I could reuse? Could someone help, pls?

Hi,

I suggest that you look at the Elasticsearch logs, you may find a more meaningful message.

Also, you can try with a full URL for the endpoint settings.

Hi,

I have the same problem. I managed the s3 plugin to connect the mock. It makes an attempt to create the snapshot but fails with error:

Unable to verify integrity of data upload. Client calculated content hash (contentMD5: Zz7VA5IqTZbUwyONkppzOQ== in base 64) didn't match hash (etag: 2fed920f2e6b4304d20e6bfd14eb32a0 in hex) calculated by Amazon S3. You may need to delete the data stored in Amazon S3. (metadata.contentMD5: null, md5DigestStream: com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream@420f87c4, bucketName: yyyrtbackup, key: tests-1OoAbPRlSD66bkuWlO3_FA/master.dat-temp

However, master.dat-temp seems to be created in s3 which is strange due to the error. My theory, why this happen is that AWS appends md5 checksum to its reply and java AWS-SDK is the only SDK which checks that checksum. It seems the s3 mock calculates invalid checksum. Our code is written in nodejs/ruby/python, so we don't have the problem accessing the mock from there.

So the problem is in Java. By default it doesn't invalidate it's DNS cache so it has no way of knowing S3 docker host. To invalidate cache I would need to start ElasticSearch with -Dnetworkaddress.cache.ttl=1 and I'm not sure I can achieve that without rewriting ElasticSearch dockerfile.

Seeing this problem and also the problem @smrt28 reported with S3 mock it seems as a better option to use real S3 bucket. Pity, it would have been a nice example of docker testing.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.