Hi,
I'm struggling to create an integration test for our code orchestrating snapshots to S3.
I've following setup:
Docker compose of external services:
services:
s3:
build: s3/
ports:
- "9495:9495"
elasticsearch:
build: elasticsearch/
ulimits:
nofile:
soft: 65536
hard: 65536
ports:
- "9000:9200"
s3 dockerfile (Should I use different s3 mock?):
FROM node:alpine
RUN npm install -g s3rver
EXPOSE 9495
CMD [ "s3rver", "--hostname", "0.0.0.0", "--port", "9495", "--directory", "/tmp" ]
elasticsearch dockerfile (I've tried various versions of this setup):
FROM docker.elastic.co/elasticsearch/elasticsearch:6.0.1
RUN bin/elasticsearch-plugin install repository-s3
RUN echo 123 | bin/elasticsearch-keystore add --stdin --force s3.client.default.secret_key
RUN echo abc | bin/elasticsearch-keystore add --stdin --force s3.client.default.access_key
RUN echo "s3.client.default.endpoint: s3:9495" >> config/elasticsearch.yml
RUN echo "s3.client.default.protocol: http" >> config/elasticsearch.yml
RUN echo "s3.client.default.proxy.host: s3" >> config/elasticsearch.yml
RUN echo "s3.client.default.proxy.port: 9495" >> config/elasticsearch.yml
Bucket creation:
aws s3api create-bucket \
--acl public-read-write \
--bucket snapshot \
--endpoint-url http://localhost:9495
When I try to register my mock bucket:
PUT _snapshot/my_s3_repository
{
"type": "s3",
"settings": {
"bucket": "snapshot",
"canned_acl": "public-read-write"
}
}
I get following answer (bucket snapshot
DOES exist):
{
"error": {
"root_cause": [
{
"type": "repository_exception",
"reason": "[my_s3_repository] failed to create repository"
}
],
"type": "repository_exception",
"reason": "[my_s3_repository] failed to create repository",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "The bucket [snapshot] does not exist. Please create it before creating an s3 snapshot repository backed by it."
}
},
"status": 500
}
With the configuration without proxy I get following answer (I'm able to ping s3 host from elasticsearch container):
{
"error": {
"root_cause": [
{
"type": "repository_exception",
"reason": "[my_s3_repository] failed to create repository"
}
],
"type": "repository_exception",
"reason": "[my_s3_repository] failed to create repository",
"caused_by": {
"type": "sdk_client_exception",
"reason": "Unable to execute HTTP request: snapshot.s3",
"caused_by": {
"type": "unknown_host_exception",
"reason": "snapshot.s3"
}
}
},
"status": 500
}
Has anyone done anything similar? Does ElasticSearch use integration test setup of their own I could reuse? Could someone help, pls?