S3 snapshot not working after ELK upgrade to 6.4.2

sreejiths · December 17, 2018, 8:58am

Hi

We have upgraded ELK from V5.6.8 to V6.4.2 . Upgrade has been completed but now i am not able to view or connect to on premises Cloud Object Storage . When i do "GET /_snapshot/" i can see the registered SNAPSHOT REPO's but when i query for details of backfiles on the REPO it gives me "repository_missing_exception"

I has updated the S3 plugin during the upgrade

Name: repository-s3
Description: The S3 repository plugin adds S3 repositories
Version: 6.4.2
Elasticsearch Version: 6.4.2
Java Version: 1.8
Native Controller: false
Extended Plugins:

Classname: org.elasticsearch.repositories.s3.S3RepositoryPlugin

Not sure what caused the breakage when upgrading to V6.4.2 ..ANyone aware of possible cause ..Dont see any specific logs also ..

dadoonet · December 17, 2018, 9:15am

What are your settings? Did you check how to define S3 settings in 6.4?
What are elasticsearch logs.

sreejiths · December 17, 2018, 9:45am

Hi David ..
Do you mean i need to set up S3 REPO again with the as per below .. My only worry is that whether i will loose the earlier SNAPSHOT back up

https://www.elastic.co/guide/en/elasticsearch/plugins/6.4/repository-s3-repository.html

sreejiths · December 17, 2018, 9:47am

ALso below is the setting i used when i created REPO in 5.6.4

{
"s3_repository": {
"type": "s3",
"settings": {
"bucket": "XXXXX",
"chunk_size": "5G",
"endpoint": "XXXXXX",
"max_retries": "6",
"port": "443",
"buffer_size": "5G"
}
}

dadoonet · December 17, 2018, 10:17am

I meant did you follow this part to create the S3 client? Client Settings | Elasticsearch Plugins and Integrations [6.4] | Elastic

Do you mean i need to set up S3 REPO again with the as per below .. My only worry is that whether i will loose the earlier SNAPSHOT back up

Creating a repository should not remove any existing backup. It's "just" a link to a storage (more or less).

"endpoint": "XXXXXX",

Would it be possible to give what the form of this XXXXXX is? It must be one of those endpoints (not region): AWS service endpoints - AWS General Reference

Then what are elasticsearch logs. You should probably see some errors at startup if the repository already exists in the cluster state.

sreejiths · December 18, 2018, 7:29am

Below is the steps that I followed ..Not sure what is happening ..Please take a look & advice

STEP 1 : Add the access key & secret key to keystore
[root@XXXX elasticsearch]# bin/elasticsearch-keystore add s3.client.default.access_key
Enter value for s3.client.default.access_key:
[root@XXXXX elasticsearch]#
[root@XXXXX elasticsearch]#
[root@XXXXX elasticsearch]# bin/elasticsearch-keystore add s3.client.default.secret_key
Enter value for s3.client.default.secret_key:
[root@x01gnwatesr7a elasticsearch]#

Error : No error

STEP 2: Create a new REPO

PUT _snapshot/Fw_Repository
{
"type": "s3",
"settings": {
"bucket": "XXXXXX",
"endpoint": "XXXXXX",
"max_retries": "20",
"chunk_size": "5G",
"buffer_size": "5G"
}
}
Error:
{
"error": {
"root_cause": [
{
"type": "repository_exception",
"reason": "[Fw_Repository] cannot create blob store"
}
],
"type": "repository_exception",
"reason": "[Fw_Repository] cannot create blob store",
"caused_by": {
"type": "sdk_client_exception",
"reason": "Unable to load credentials from service endpoint",
"caused_by": {
"type": "socket_timeout_exception",
"reason": "connect timed out"
}
}
},
"status": 500
}
Note : No connection issue with S3 endpoint

Step 3 : verification of S3 repo created

GET /_snapshot/Fw_Repository

It seems like Repo is created eventhough error was seen is step 2
Output:
{
"Fw_Repository": {
"type": "s3",
"settings": {
"bucket": "sg00tsimnwtsg",
"chunk_size": "5G",
"endpoint": "s3.sgp.dbs.com",
"max_retries": "20",
"buffer_size": "5G"
}
}
}

Step 4: Trying to create a snapshot

PUT /_snapshot/Fw_Repository/snapshot_test?wait_for_completion=true
{
"indices": "logstash-nts-2018.12.18",
"ignore_unavailable": true,
"include_global_state": false
}

Output:
{
"error": {
"root_cause": [
{
"type": "repository_exception",
"reason": "[Fw_Repository] cannot create blob store"
}
],
"type": "repository_exception",
"reason": "[Fw_Repository] cannot create blob store",
"caused_by": {
"type": "sdk_client_exception",
"reason": "Unable to load credentials from service endpoint",
"caused_by": {
"type": "socket_timeout_exception",
"reason": "connect timed out"
}
}
},
"status": 500
}

dadoonet · December 18, 2018, 8:01am

2 things:

did you restart?
"endpoint": "s3.sgp.dbs.com", is not a valid S3 endpoint so I don't think it's supported.

sreejiths · December 18, 2018, 10:01am

did you restart?
Yes
"endpoint": "s3.sgp.dbs.com", is not a valid S3 endpoint so I don't think it's supported.

We are using on-premises Cloud Object Storage and not AWS ..

As per the error i think issue the by default its trying to connect to AWS and not to our On premises S3 endpoint ..How to fix it

[root@XXXXX support]# curl -XGET -u elastic -k "https://localhost:9200/_snapshot/Fw_Repository/snapshot_test?pretty"
Enter host password for user 'elastic':
{
"error" : {
"root_cause" : [
{
"type" : "repository_exception",
"reason" : "[Fw_Repository] cannot create blob store"
}
],
"type" : "repository_exception",
"reason" : "[Fw_Repository] cannot create blob store",
"caused_by" : {
"type" : "sdk_client_exception",
"reason" : "Unable to execute HTTP request: Connect to sg00tsimnwtsg.s3.amazonaws.com:443 [sg00tsimnwtsg.s3.amazonaws.com/52.216.82.24] failed: connect timed out",
"caused_by" : {
"type" : "connect_timeout_exception",
"reason" : "Connect to sg00tsimnwtsg.s3.amazonaws.com:443 [sg00tsimnwtsg.s3.amazonaws.com/52.216.82.24] failed: connect timed out",
"caused_by" : {
"type" : "socket_timeout_exception",
"reason" : "connect timed out"
}
}
}
},
"status" : 500
}

dadoonet · December 18, 2018, 10:53am

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.

Here I believe that your elasticsearch node can not access to sg00tsimnwtsg.s3.amazonaws.com on port 443. May be check some firewall rules?

Also activate may be the DEBUG mode for the following packages:

org.elasticsearch.repositories.s3
com.amazon

sreejiths · December 19, 2018, 12:44am

David

First of all thanks for help and suggestions ...

We are using on-premises Cloud Object Storage and not AWS S3 ..

We have already upgraded our Production cluster also to V6.4.2 and now we are stuck as we were heavily relying on S3 snaphot on on-premises S3 custom storage (Cleversafe/IBM S3 storage ) .

I believe the elasticsearch S3 plugin by default is calling the amazonaws S3 and not the endpoint specified in the REPO config specified , any temporary fix ELASTIC can provide to fix this default behaviour ??

ERROR

"type" : "repository_exception",
"reason" : "[Fw_Repository] cannot create blob store",
"caused_by" : {
"type" : "sdk_client_exception",
"reason" : "Unable to execute HTTP request: Connect to sg00tsimnwtsg.s3.amazonaws.com:443[sg00tsimnwtsg.s3.amazonaws.com/52.216.82.24] failed: connect timed out",
"caused_by" : {
"type" : "connect_timeout_exception",
"reason" : "Connect to sg00tsimnwtsg.s3.amazonaws.com:443[sg00tsimnwtsg.s3.amazonaws.com/52.216.82.24] failed: connect timed out",
"caused_by" : {
"type" : "socket_timeout_exception",
"reason" : "connect timed out"
}
}
}
},
"status" : 500
}

dadoonet · December 19, 2018, 1:10am

Please format your logs as I asked before.

Could you create a new repository instead of Fw_Repository?
And paste exactly all what you did and the full response?

I'm surprised that Fw_Repository exists although it has been rejected.

sreejiths · December 19, 2018, 2:10am

Hi David

Will create a new REPO and update asap..just to make things clear please check below

As per below elastic documentation as below specific for "S3 endpoint configuration" in 6.4.2 , S3 endpoint will be automatically figured out based on AWS bucket location ..Any way we can differ from this behavior . As we are using ON-PREMISE S3 storage and NOT AWS , the S3 connection by default is been pushed to AWS and not to actual on premises S3 endpoint ..

https://www.elastic.co/guide/en/elasticsearch/plugins/6.4/repository-s3-client.html

Again Thanks for your time ..

dadoonet · December 19, 2018, 6:01am

No the documentation is wrong.

system · January 16, 2019, 6:02am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.