Restoring snapshot (filesystem) - empty

Elastic Stack Elasticsearch
1

Hi,
TL;DR;
Tried to restore a snapshot I did before re-installing the machine but ended up with elasticsearch complaining the snapshot is "empty"

What happened:
I had to set up a machine from scratch (single instance ELK installation, no cluster running under Ubuntu). So I created a snapshot, tar-ed the resulting directory and put it in a safe place on another machine. After setting up the machine (OS, ELK 8.17.0) I wanted to restore the snapshot only to find out that elasticsearch.

I "tar xpf"-ed the snapshot tar (~31GB) I created on the "old" installation (also Ubuntu), made sure it's owned by the elasticsearch-user:

drwxr-xr-x 3 elasticsearch elasticsearch 4096 Nov 14 12:23 Zappy-snapshot-Repo

Next I added the path the directory is in to /etc/elasticsearch/elasticsearch.yml, i.e.

path.repo: /tmp/Zappy-snapshot-Repo

Next I went to kibana and added the Repo. No errors here, but I got a message that the Repo has no snapshots:

image
image551×465 19 KB

Same in the developer console

GET _snapshot/

gives

{
  "My-Test-Repo": {
    "type": "fs",
    "settings": {
      "location": "/tmp/Zappy-snapshot-Repo"
    }
  }
}

but

GET _snapshot/_status

gives

{
  "snapshots": []
}

i.e. "empty" :thinking:

No error messages whatsoever in /var/log/elasticsearch/elasticsearch.log

Also tried with debuging, i.e.

PUT _cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.snapshots": "DEBUG"
  }
}

Still no useful information in elasticsearch.log.

Any ideas what could be wrong, i.e. why I can't re-import the snapshot?

2

What do you have inside of /tmp/Zappy-snapshot-Repo?

Please run a ls on that path and share the results.

3

Here we go:

/tmp/Zappy-snapshot-Repo# ls -la
total 204
drwxr-xr-x   3 elasticsearch elasticsearch   4096 Nov 14 12:23 .
drwxrwxrwt  17 root          root            4096 Jan  9 15:30 ..
-rw-r--r--   1 elasticsearch elasticsearch  40680 Nov 14 12:23 index-0
-rw-r--r--   1 elasticsearch elasticsearch      8 Nov 14 12:23 index.latest
drwxr-xr-x 149 elasticsearch elasticsearch  12288 Nov 14 12:14 indices
-rw-r--r--   1 elasticsearch elasticsearch 135746 Nov 14 12:23 meta-6bs6VgeaQ7-StvlUpmov1g.dat
-rw-r--r--   1 elasticsearch elasticsearch   3224 Nov 14 12:23 snap-6bs6VgeaQ7-StvlUpmov1g.dat
/tmp/Zappy-snapshot-Repo#

so it's owned by "elasticsearch:elasticsearch" with the exact same permissions as the original (tar-ed via "tar cf" and extracted with "tar xpf")

4

What happens when you click in Verify Repository ?

From the documentation, it suggests that you need to Verify the repository before starting to use it.

5

No complaints, just the message that the repo has no snapshots.

In the meantime though thanks to a tip from one of my colleagues I could fix the problem: I originally put the directory containing the snapshot files to /tmp where elasticsearch didn't find them. As soon as I moved the files making up the repo away from /tmp to some other place in the filesystem elasticsearch was able to access them. Seems to have something to do with the way systemd treats /tmp.

6

Yes, Elasticsearch sets the PrivateTmp option by default. You can adjust this if you really need to, but the simplest solution is to avoid putting non-temporary (or at least non-process-private) stuff like snapshots in /tmp.