Adding S3 bucket for snapshot

Elastic Stack Elasticsearch
1

I am trying to add S3 bucket for taking snapshot of my elastic cluster. I am running Elasticsearch 2.3.3 and installed plugin cloud-aws. Following is syntax of my request

curl -X POST "http://XXX.XXX.XXX.XXX:9200/_snapshot/s3_repository" -d '{"type": "s3","settings":{"bucket":"bucketname","region":"ap-south-1","access_key":"accesskey","secret_key":"secretkey","endpoint":"s3.ap-south-1.amazonaws.com","compress":true}}'

it gives me following error

{"error":{"root_cause":[{"type":"repository_exception","reason":"[s3_repository] failed to create repository"}],"type":"repository_exception","reason":"[s3_repository] failed to create repository","caused_by":{"type":"creation_exception","reason":"Guice creation errors:\n\n1) Error injecting constructor, com.amazonaws.AmazonClientException: Unable to execute HTTP request: bucketname.s3.ap-south-1.amazonaws.com\n at org.elasticsearch.repositories.s3.S3Repository.(Unknown Source)\n while locating org.elasticsearch.repositories.s3.S3Repository\n while locating org.elasticsearch.repositories.Repository\n\n1 error","caused_by":{"type":"amazon_client_exception","reason":"Unable to execute HTTP request: bucketname.s3.ap-south-1.amazonaws.com","caused_by":{"type":"unknown_host_exception","reason":"bucketname.s3.ap-south-1.amazonaws.com"}}}},"status":500}

2

If you are registering a snapshot, then it's a PUT, not a POST. I think
this document explains it well.

3

Thanks for response , tried with PUT but getting same exception/error.

4

Hi gaurav...i am new to kibana..plz can u provide links about aggregations..i dont understand where should i load json file..i dont know how to generate the charts by passing the aggregations i am looking this url but i dont understand how to implement..https://www.elastic.co/guide/en/elasticsearch/guide/current/_returning_empty_buckets.html ..can u help me

5

@venkat545 don’t hijack threads like this and open your own questions. Your answer is totally unrelated.

6

@dadoonet Please help to resolve the issue , have gone through documentation and did not found any solution to my problem.

7

@mudgilgaurav please read this and specifically the "Also be patient" part.

Do you have a more detailed stack-trace in your logs?

Judging from the error it seems that http://bucketname.s3.ap-south-1.amazonaws.com/ does not exist. Did you create it?

Also, in this old version, if you define the region you don't need to define the endpoint.
In 6.0 you need to define the endpoint only as region has been removed.

Any chance you can also upgrade? At least to latest 2.4?

8
  • Does the bucket exist?
  • Does the machine from where you are running the snapshot have adequate permissions to write to the bucket?
9

@dadoonet Yes this bucket exists. I created the bucket and did not modified any permission on it by default bucket admin can perform any operation. Access key and secret used for admin user only.

I tried with region only but got following error

{"error":{"root_cause":[{"type":"repository_exception","reason":"[s3_repository] failed to create repository"}],"type":"repository_exception","reason":"[s3_repository] failed to create repository","caused_by":{"type":"creation_exception","reason":"Guice creation errors:\n\n1) Error injecting constructor, java.lang.IllegalArgumentException: No automatic endpoint could be derived from region [ap-south-1]\n at org.elasticsearch.repositories.s3.S3Repository.(Unknown Source)\n while locating org.elasticsearch.repositories.s3.S3Repository\n while locating org.elasticsearch.repositories.Repository\n\n1 error","caused_by":{"type":"illegal_argument_exception","reason":"No automatic endpoint could be derived from region [ap-south-1]"}}},"status":500}

Currently we are migration our application to AWS so wants to do only lift and shift , version upgrade can be done once we are migrated to AWS.

10

@mudgilgaurav

I don't think thats a region. Its either ap-southeast-1 or ap-southeast-2 ...

11

Actually, my bad. That is a region! :frowning: Mumbai!

12

@dadoonet I configured aws-cli on my local server and configured my credentials. I am able to list all buckets created by me. While during adding s3 repository to elasticsearch still getting same error what is posted initially.

13

This error No automatic endpoint could be derived from region [ap-south-1] is because you are using an old version.
But anyway, remove the region field entirely and only use the endpoint.

I configured aws-cli on my local server and configured my credentials. I am able to list all buckets created by me.

Great. Can you do the same thing from the server where elasticsearch is running?

Currently we are migration our application to AWS so wants to do only lift and shift , version upgrade can be done once we are migrated to AWS.

Just a note about this in case you'd like to simplify your management of your elastic platform, just use cloud.elastic.co. You'll get so many included features like X-Pack, Kibana, Automatic backups, Network Bandwidth usage included...

14

Yes I can do it from server where elasticsearch is running.

Yes cloud.elastic.co could be good option , we will explore AWS elasticsearch vs cloud.elastic.co post migration to AWS.

Is my endpoint correct i.e bucketname.s3.ap-south-1.amazonaws.com??

Is my endpoint correct i.e .s3.ap-south-1.amazonaws.com??

If i am able to access my bucket from elasticsearch server then I do not know why still getting unknown host error.

15

Is my endpoint correct i.e .s3.ap-south-1.amazonaws.com

The correct endpoint should be s3.ap-south-1.amazonaws.com.

If i am able to access my bucket from elasticsearch server then I do not know why still getting unknown host error.

Could you change the log level for org.elasticsearch.cloud.aws to DEBUG?

16

My endpoint is correct.

For changing the log level of org.elasticsearch.cloud.aws to debug do i need to update and redeploy plugins/cloud-aws/commons-logging-1.1.3.jar or I can do it through elasticsearch.yml?

17

You can change it live. See https://www.elastic.co/guide/en/elasticsearch/reference/2.3/misc-cluster.html#cluster-logger

18

@dadoonet I enabled the suggested log level , please fine elasticsearch log below

[2017-12-09 19:57:57,474][WARN ][rest.suppressed ] /_snapshot/s3_repository Params: {repository=s3_repository}

RepositoryException[[s3_repository] failed to create repository]; nested: CreationException[Guice creation errors:

  1. Error injecting constructor, com.amazonaws.AmazonClientException: Unable to execute HTTP request: bucketname-snapshot.s3.ap-south-1.amazonaws.com

at org.elasticsearch.repositories.s3.S3Repository.(Unknown Source)

while locating org.elasticsearch.repositories.s3.S3Repository

while locating org.elasticsearch.repositories.Repository

1 error]; nested: AmazonClientException[Unable to execute HTTP request: bucketname-snapshot.s3.ap-south-1.amazonaws.com]; nested: UnknownHostException[bucketname-snapshot.s3.ap-south-1.amazonaws.com];

at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:411)
19

I'm expecting more logs than that. Could you share all the logs?
How many nodes do you have? If more than one, could you check in the other machines?

And please format your logs with the </>. The preview window will tell you when it's correct or not.

20
  1. Error injecting constructor, com.amazonaws.AmazonClientException: Unable to execute
    HTTP request: b-snapshot.s3.ap-south-1.amazonaws.com
    at org.elasticsearch.repositories.s3.S3Repository.(Unknown Source)
    while locating org.elasticsearch.repositories.s3.S3Repository
    while locating org.elasticsearch.repositories.Repository
    1 error]; nested: AmazonClientException[Unable to execute HTTP request: b-
    snapshot.s3.ap-south-1.amazonaws.com]; nested: UnknownHostException[b- snapshot.s3.ap-south-1.amazonaws.com]; at
    org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:411)
    atorg.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:368)
    at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:55)
    at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:110)
    atorg.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468) at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231) at org.elas ticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748) Caused by: org.elasticsearch.common.inject.CreationException: Guice creation errors:
    1. Error injecting constructor, com.amazonaws.AmazonClientException: Unable to execute HTTP request: b-snapshot.s3.ap-south-1.amazonaws.com at org.elasticsearch.repositories.s3.S3Repository.(Unknown Source)
      while locating org.elasticsearch.repositories.s3.S3Repository
      while locating org.elasticsearch.repositories.Repository
      1 error at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:360) at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
      at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
      at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:157)
      at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
      at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
      ... 11 more
      Caused by: com.amazonaws.AmazonClientException: Unable to execute HTTP request: 91review-snapshot.s3.ap-south-1.amazonaws.com
      at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:517)
      at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:318)
      at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595)
      at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1041)
      at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1013)
      at org.elasticsearch.cloud.aws.blobstore.S3BlobStore.(S3BlobStore.java:84)
      at org.elasticsearch.repositories.s3.S3Repository.(S3Repository.java:132)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:50)
      at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
      at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
      at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:54)
      at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
      at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)
      at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
      at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
      at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)
      at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:201)
      at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
      at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:879)
      at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193) at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
      ... 15 more
      Caused by: java.net.UnknownHostException: 91review-snapshot.s3.ap-south-
      1.amazonaws.com
      at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
      at java.net.InetAddress.getAllByName(InetAddress.java:1192)
      at java.net.InetAddress.getAllByName(InetAddress.java:1126)
      at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
      at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) atorg.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:259) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:159) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446) . at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
      at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:769)
      at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:506)
      ... 39 more