# Failed to create S3 repository with ES 6.0 **URL:** \ **Category:** Elasticsearch\ **Created:** [November 18, 2017, 6:47am UTC](https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224 "2017-11-18T06:47:12Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![outcoldman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/outcoldman/32/20061_2.png) [@outcoldman](https://discuss.elastic.co/u/outcoldman)\ **Post date:** [November 18, 2017, 6:47am UTC](https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224/1 "2017-11-18T06:47:12Z")
I used S3 repository with ES 5 fine without any issue. Used ``` curl -s -XPUT 127.0.0.1:9200/_snapshot/s3 -d "{ \"type\": \"s3\", \"settings\": { \"bucket\": \"BUCKET_NAME\", \"region\": \"us-west-2\", \"base_path\": \"elasticsearch/\", \"access_key\": \"$(cat /aws/access_key)\", \"secret_key\": \"$(cat /aws/secret_key)\", \"compress\": true, \"server_side_encryption\": true } }" ``` In ES 6 now you need to put access\_key and secret\_key inside keystore, so now I do ``` cat /aws/access_key | bin/elasticsearch-keystore add --stdin --force s3.client.default.access_key cat /aws/secret_key | bin/elasticsearch-keystore add --stdin --force s3.client.default.secret_key ``` And after that ``` curl -s -XPUT -H "Content-Type: application/json" 127.0.0.1:9200/_snapshot/s3 -d "{ \"type\": \"s3\", \"settings\": { \"bucket\": \"BUCKET_NAME\", \"base_path\": \"elasticsearch6/\", \"compress\": true, \"server_side_encryption\": true } }" ``` But that command fails with ``` {"error":{"root_cause":[{"type":"repository_exception","reason":"[s3] failed to create repository"}],"type":"repository_exception","reason":"[s3] failed to create repository","caused_by":{"type":"sdk_client_exception","reason":"Unable to load credentials from service endpoint","caused_by":{"type":"socket_timeout_exception","reason":"connect timed out"}}},"status":500} ``` Not sure if I am missing something or this is a bug. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 18, 2017, 7:43am UTC](https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224/2 "2017-11-18T07:43:05Z")
Is there more logs like a stacktrace? Could you change the log level? ---
**Author:** ![outcoldman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/outcoldman/32/20061_2.png) [@outcoldman](https://discuss.elastic.co/u/outcoldman)\ **Post date:** [November 19, 2017, 8:31pm UTC](https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224/3 "2017-11-19T20:31:59Z")
@dadoonet yep, I have them, wanted to pasted it with the original question, but there is a limit for the message size. this is a link to gist with what I see in logs [https://gist.github.com/outcoldman/2ff70c8e952d7d61d03c2cc6cbc4cd83](https://gist.github.com/outcoldman/2ff70c8e952d7d61d03c2cc6cbc4cd83) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 19, 2017, 8:48pm UTC](https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224/4 "2017-11-19T20:48:12Z")
Apparently this is caused by: ```auto Caused by: java.net.SocketTimeoutException: connect timed out ``` Are you running this curl call from an EC2 instance or locally from your laptop? ---
**Author:** ![outcoldman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/outcoldman/32/20061_2.png) [@outcoldman](https://discuss.elastic.co/u/outcoldman)\ **Post date:** [November 19, 2017, 9:15pm UTC](https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224/5 "2017-11-19T21:15:19Z")
@dadoonet it is a server at home, btw I also enabled DEBUG logging and found ``` [2017-11-19T21:12:23,435][DEBUG][o.e.r.s.InternalAwsS3Service] [elasticsearch] creating S3 client with client_name [default], endpoint [] [2017-11-19T21:12:23,435][DEBUG][o.e.r.s.InternalAwsS3Service] [elasticsearch] Using instance profile credentials [2017-11-19T21:12:23,439][DEBUG][c.a.s.s.AmazonS3Client] Bucket region cache doesn't have an entry for outcold.net-backup. Trying to get bucket region from Amazon S3. ``` Which means my credentials aren't used. But ``` # elasticsearch-keystore list keystore.seed s3.client.default.access_key s3.client.default.secret_key ``` I assume maybe bug somewhere in documentation? Do I need to do anything else to make these keystore records visible for S3 plugin? ---
**Author:** ![outcoldman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/outcoldman/32/20061_2.png) [@outcoldman](https://discuss.elastic.co/u/outcoldman)\ **Post date:** [November 19, 2017, 9:21pm UTC](https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224/6 "2017-11-19T21:21:00Z")
@dadoonet looking on the code [https://github.com/elastic/elasticsearch/blob/17d01550c2d8736c3ed38c5eeface1d292c70717/plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/InternalAwsS3Service.java#L112](https://github.com/elastic/elasticsearch/blob/17d01550c2d8736c3ed38c5eeface1d292c70717/plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/InternalAwsS3Service.java#L112) it does not seem like this code actually loads these values from keystore. It does tell you about deprecation, but keeps using the `SecureSetting.insecureString`, where as I understand it should use `SecureSetting.secureString` to have access to the keystore, right? ---
**Author:** ![outcoldman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/outcoldman/32/20061_2.png) [@outcoldman](https://discuss.elastic.co/u/outcoldman)\ **Post date:** [November 19, 2017, 9:41pm UTC](https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224/7 "2017-11-19T21:41:23Z")
And I have another guess what can cause it (if I am not right about that this code does not read from the keystore). With docker (in my case kubernetes) I keep only data folder as stateful. Also I add the keys into keystore after my ES is already started. So there are two issues: 1. Looking on [https://github.com/elastic/elasticsearch/issues/26589](https://github.com/elastic/elasticsearch/issues/26589) - seems like these settings should be set before you start ES. 2. Because of Docker environments - keystore is stored near elasticsearch.yml file, which means that I do not keep it between restarts. As a workaround I assume I can use `es.allow_insecure_settings` and use old way, but it does not seems like a documented flag, not sure where I can set it. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 19, 2017, 9:53pm UTC](https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224/8 "2017-11-19T21:53:01Z")
Secured Settings are read from this class > ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 19, 2017, 9:53pm UTC](https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224/9 "2017-11-19T21:53:31Z")
Could you try without Docker? ---
**Author:** ![outcoldman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/outcoldman/32/20061_2.png) [@outcoldman](https://discuss.elastic.co/u/outcoldman)\ **Post date:** [November 19, 2017, 10:22pm UTC](https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224/10 "2017-11-19T22:22:51Z")
Yep, so I can confirm, that because these settings aren't dynamic you cannot modify them after you already started the ElasticSearch in docker. Considering that config folder is not a state folder, that makes it very problematic. As a workaround I am pre-building the keystore in my image ``` FROM docker.elastic.co/elasticsearch/elasticsearch-basic:6.0.0 RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3 RUN mkdir -p /aws COPY access_key /aws/access_key COPY secret_key /aws/secret_key RUN \ cat /aws/access_key | base64 --decode | bin/elasticsearch-keystore add --stdin --force s3.client.default.access_key && \ cat /aws/secret_key | base64 --decode | bin/elasticsearch-keystore add --stdin --force s3.client.default.secret_key RUN rm -fR /aws ``` But there are a lot of issues with that: 1. Why elasticsearch docker image actually have a keystore prebuild in the image? Is that a security bug? I do not know how it is implemented, but assuming that all the users on this image basically can just steel the keystore and use in their own image? 2. Now I am required to pass the AWS credentials on build step. And as I recall there are no secure way to do that with docker. Before 6 I used kubernetes security store to manage my secrets, which is a little bit more secure. Plus I don't need to store them in the keystore with the image. @dadoonet Could you please pass this feedback to the people who maintain docker image? I can provide additional feedback if that will be required. And thank you for helping. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 20, 2017, 2:16am UTC](https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224/11 "2017-11-20T02:16:24Z")
Cc @rjernst I think that in the future we will be able to update secured settings while the node is running but I’m not sure about the plan for this. Thanks for sharing the workaround BTW. May be worth writing it in the [secure settings documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-settings.html)? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 18, 2017, 2:16am UTC](https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224/12 "2017-12-18T02:16:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.