Failed to create S3 repository with ES 6.0

outcoldman · November 18, 2017, 6:47am

I used S3 repository with ES 5 fine without any issue. Used

curl -s -XPUT 127.0.0.1:9200/_snapshot/s3 -d "{
              \"type\": \"s3\",
              \"settings\": {
                \"bucket\": \"BUCKET_NAME\",
                \"region\": \"us-west-2\",
                \"base_path\": \"elasticsearch/\",
                \"access_key\": \"$(cat /aws/access_key)\",
                \"secret_key\": \"$(cat /aws/secret_key)\",
                \"compress\": true,
                \"server_side_encryption\": true
              }
            }"

In ES 6 now you need to put access_key and secret_key inside keystore, so now I do

cat /aws/access_key | bin/elasticsearch-keystore add --stdin --force s3.client.default.access_key
cat /aws/secret_key | bin/elasticsearch-keystore add --stdin --force s3.client.default.secret_key

And after that

curl -s -XPUT -H "Content-Type: application/json" 127.0.0.1:9200/_snapshot/s3 -d "{
                  \"type\": \"s3\",
                  \"settings\": {
                    \"bucket\": \"BUCKET_NAME\",
                    \"base_path\": \"elasticsearch6/\",
                    \"compress\": true,
                    \"server_side_encryption\": true
                  }
                }"

But that command fails with

{"error":{"root_cause":[{"type":"repository_exception","reason":"[s3] failed to create repository"}],"type":"repository_exception","reason":"[s3] failed to create repository","caused_by":{"type":"sdk_client_exception","reason":"Unable to load credentials from service endpoint","caused_by":{"type":"socket_timeout_exception","reason":"connect timed out"}}},"status":500}

Not sure if I am missing something or this is a bug.

dadoonet · November 18, 2017, 7:43am

Is there more logs like a stacktrace?

Could you change the log level?

outcoldman · November 19, 2017, 8:31pm

@dadoonet yep, I have them, wanted to pasted it with the original question, but there is a limit for the message size.

this is a link to gist with what I see in logs https://gist.github.com/outcoldman/2ff70c8e952d7d61d03c2cc6cbc4cd83

dadoonet · November 19, 2017, 8:48pm

Apparently this is caused by:

Caused by: java.net.SocketTimeoutException: connect timed out

Are you running this curl call from an EC2 instance or locally from your laptop?

outcoldman · November 19, 2017, 9:15pm

@dadoonet it is a server at home, btw I also enabled DEBUG logging and found

[2017-11-19T21:12:23,435][DEBUG][o.e.r.s.InternalAwsS3Service] [elasticsearch] creating S3 client with client_name [default], endpoint []
[2017-11-19T21:12:23,435][DEBUG][o.e.r.s.InternalAwsS3Service] [elasticsearch] Using instance profile credentials
[2017-11-19T21:12:23,439][DEBUG][c.a.s.s.AmazonS3Client   ] Bucket region cache doesn't have an entry for outcold.net-backup. Trying to get bucket region from Amazon S3.

Which means my credentials aren't used.

But

# elasticsearch-keystore list  
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key

I assume maybe bug somewhere in documentation? Do I need to do anything else to make these keystore records visible for S3 plugin?

outcoldman · November 19, 2017, 9:21pm

@dadoonet looking on the code https://github.com/elastic/elasticsearch/blob/17d01550c2d8736c3ed38c5eeface1d292c70717/plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/InternalAwsS3Service.java#L112 it does not seem like this code actually loads these values from keystore. It does tell you about deprecation, but keeps using the SecureSetting.insecureString, where as I understand it should use SecureSetting.secureString to have access to the keystore, right?

outcoldman · November 19, 2017, 9:41pm

And I have another guess what can cause it (if I am not right about that this code does not read from the keystore).

With docker (in my case kubernetes) I keep only data folder as stateful. Also I add the keys into keystore after my ES is already started. So there are two issues:

Looking on https://github.com/elastic/elasticsearch/issues/26589 - seems like these settings should be set before you start ES.
Because of Docker environments - keystore is stored near elasticsearch.yml file, which means that I do not keep it between restarts.

As a workaround I assume I can use es.allow_insecure_settings and use old way, but it does not seems like a documented flag, not sure where I can set it.

dadoonet · November 19, 2017, 9:53pm

Secured Settings are read from this class

github.com

elastic/elasticsearch/blob/17d01550c2d8736c3ed38c5eeface1d292c70717/plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3ClientSettings.java

/*
 * Licensed to Elasticsearch under one or more contributor
 * license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright
 * ownership. Elasticsearch licenses this file to you under
 * the Apache License, Version 2.0 (the "License"); you may
 * not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */

package org.elasticsearch.repositories.s3;

This file has been truncated. show original

dadoonet · November 19, 2017, 9:53pm

Could you try without Docker?

outcoldman · November 19, 2017, 10:22pm

Yep, so I can confirm, that because these settings aren't dynamic you cannot modify them after you already started the ElasticSearch in docker. Considering that config folder is not a state folder, that makes it very problematic.

As a workaround I am pre-building the keystore in my image

FROM docker.elastic.co/elasticsearch/elasticsearch-basic:6.0.0

RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3

RUN mkdir -p /aws

COPY access_key /aws/access_key
COPY secret_key /aws/secret_key

RUN \
  cat /aws/access_key | base64 --decode | bin/elasticsearch-keystore add --stdin --force s3.client.default.access_key && \
  cat /aws/secret_key | base64 --decode | bin/elasticsearch-keystore add --stdin --force s3.client.default.secret_key

RUN rm -fR /aws

But there are a lot of issues with that:

Why elasticsearch docker image actually have a keystore prebuild in the image? Is that a security bug? I do not know how it is implemented, but assuming that all the users on this image basically can just steel the keystore and use in their own image?
Now I am required to pass the AWS credentials on build step. And as I recall there are no secure way to do that with docker. Before 6 I used kubernetes security store to manage my secrets, which is a little bit more secure. Plus I don't need to store them in the keystore with the image.

@dadoonet Could you please pass this feedback to the people who maintain docker image? I can provide additional feedback if that will be required. And thank you for helping.

dadoonet · November 20, 2017, 2:16am

Cc @rjernst

I think that in the future we will be able to update secured settings while the node is running but I’m not sure about the plan for this.

Thanks for sharing the workaround BTW. May be worth writing it in the secure settings documentation?

system · December 18, 2017, 2:16am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Is repository-s3 plugin fixed in 6.3? Elasticsearch	5	3055	July 25, 2018
S3 repository Snapshot fails on ES 6.2 Elasticsearch	4	2647	April 18, 2018
Elasticsearch \| S3 Repository and Snapshot Creation Failure Elasticsearch	13	3875	April 9, 2019
Problem creating a s3 repository with Elasticsearch Elasticsearch	1	325	April 11, 2019
S3 Repository Creation Elasticsearch	4	2006	July 6, 2017

Failed to create S3 repository with ES 6.0

Related topics