Restoring from snapshot locally "Some repositories contain errors"

[Kay_Khan]

Hi,

I have setup a local version of elasticsearch and kibana.

I want to restore an index from production to this local version.

I have created an s3 "repository" under Stack Management > Snapshot and Restore.

After creating the repository when i go to the snapshot tab i can see the following messgaes. "Some repositories contain errors, Go to repositories to fix the errors." but in production this usually shows all the snapshots.

When i go back to the repository i just created like it suggested, there is no indication of any errors.

elasticsearch/kibana version: 7.9.3

[Yulia_Cech]

Hi @Kay_Khan,

it seems that Kibana can't load your repository information on the snapshots tab. Could you please paste what the request 'snapshots' returns in Network tab of Dev tools?
(To access Dev tools, open Snapshots tab in Kibana, right click anywhere on the page and select 'Inspect'. In the Dev tools window, select 'Network' tab and reload 'Snapshots' tab. Then select 'snapshots' request)

[Kay_Khan]

@Yulia_Cech

Please see screenshot below of

This suggests to me the message "Some repositories contain errors" is kind of a misleading message.

Why is the request timing out on this local version but on production its fine?

30000ms/30s is a little too quick to timeout, what if the backup contains many snapshots?

update:

on production the request to get snapshots takes 4 seconds, so i do believe 30 seconds should be enough. But for some reason locally it is having trouble requesting for snapshots and im not sure why

[Kay_Khan]

Local Setup:

version: '3.7'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.type=single-node
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - "9200:9200"
    command: >
      /bin/sh -c "./bin/elasticsearch-plugin list | grep -q repository-s3
      || ./bin/elasticsearch-plugin install --batch repository-s3;
      echo "<redacted>" | bin/elasticsearch-keystore add --stdin --force s3.client.default.access_key;
      echo "<redacted>"| bin/elasticsearch-keystore add --stdin --force s3.client.default.secret_key;
      /usr/local/bin/docker-entrypoint.sh"
  kibana:
    image: docker.elastic.co/kibana/kibana:7.9.2
    ports:
      - "5601:5601"

There are no clear error that appears in the docker container logs

You can see the response log from trying to retrieve snapshots

kibana_1         | {"type":"response","@timestamp":"2021-02-15T16:38:43Z","tags":[],"pid":7,"method":"get","statusCode":200,"req":{"url":"/api/snapshot_restore/snapshots","method":"get","headers":{"host":"localhost:5601","connection":"keep-alive","kbn-version":"7.9.2","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36","content-type":"application/json","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/app/management/data/snapshot_restore/snapshots","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"172.21.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36","referer":"http://localhost:5601/app/management/data/snapshot_restore/snapshots"},"res":{"statusCode":200,"responseTime":30018,"contentLength":9},"message":"GET /api/snapshot_restore/snapshots 200 30018ms - 9.0B"}

[Yulia_Cech]

Hi @Kay_Khan,

you could use Dev Tools -> Console in Kibana to update repository's timeout setting with a command similar to this

There is a link to a docs section.

[Kay_Khan]

@Yulia_Cech

This does not work i modified yours slightly to fix with my repo type.

PUT _snapshot/aws-s3-repo
{
  "timeout": "300s",
  "type" : "s3",
  "settings" : {
    "bucket" : "<redacted>",
    "readonly" : "true",
    "client" : "default"
  }
}

I recieved an ack response

{
  "acknowledged" : true
}

However, the snapshots request is still timing out after 30 seconds.