Snapshot Error

Elastic Stack Elasticsearch
1

Hi All,

I have been facing issue while taking snapshot of the indices in elasticsearch cluster.

Command : curl -XPUT localhost:9200/_snapshot/first_1 -d '{
"type": "fs",
"settings": {
"location": "/tmp/snapshot_production",
"compress": true,
"chunk_size": "10m"
}
}'

Error:

{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[first_1] [QJyo4WJ5Ry-d0U4myaALgw, 'RemoteTransportException[[Tri-Man][10.0.0.1:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[first_1] store location [/tmp/snapshot_production] is not accessible on the node [{Tri-Man}{QJyo4WJ5Ry-d0U4myaALgw}{10.0.0.1}{10.0.0.1:9300}]]; nested: NotSerializableExceptionWrapper[access_denied_exception: /tmp/snapshot_production/tests-XTHsgu0rRS6WDxgnUpycyw/data-QJyo4WJ5Ry-d0U4myaALgw.dat];']]"}],"type":"repository_verification_exception","reason":"[first_1] [QJyo4WJ5Ry-d0U4myaALgw, 'RemoteTransportException[[Tri-Man][10.0.0.1:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[first_1] store location [/tmp/snapshot_production] is not accessible on the node [{Tri-Man}{QJyo4WJ5Ry-d0U4myaALgw}{10.0.0.1}{10.0.0.1:9300}]]; nested: NotSerializableExceptionWrapper[access_denied_exception: /tmp/snapshot_production/tests-XTHsgu0rRS6WDxgnUpycyw/data-QJyo4WJ5Ry-d0U4myaALgw.dat];']]"},"status":500}

Server1: 10.0.0.0
Server2: 10.0.0.1

Note:

  1. Added both machine as elasticsearch cluster. But i have been facing the above error while taking the snapshot.
    Also i have followed all your solution for this error. But still i am unable to solve this issue.

  2. Attached the output of the following command

Command : ls -ld /tmp/snapshot_production/

drwxrwxrwx. 2 root root 42 Dec 13 12:14 /tmp/snapshot_production/

Please guide me in solving this issue.

Thanks,
Gogul

2

I doubt /tmp is a shared folder by both machines. It is?

3

Hi Dadoonet,

I have mounted the shared folder in "/tmp/snapshot" of both machines.

Thanks,
Gogul

4

But you are mentioning /tmp/snapshot_production/?

5

HI Dadoonet,

Yes, I have mounted the shared path of "nfs-server" only in this location. In both Elasticsearch server.

Server1:

$ df -h

Filesystem                                 Size  Used Avail Use% Mounted on
/dev/sda4                                  329G  9.3G  320G   3% /
nfs.server12.io:/snapshot                  329G  9.3G  320G   3% /tmp/snapshot_production

Server2 :

$ df -h

Filesystem                                    Size  Used Avail Use% Mounted on
/dev/sda2                                     329G  9.4G  320G   3% /
nfs.server12.io:/snapshot                     329G  9.3G  320G   3% /tmp/snapshot_production
6

I'm not an ops guy so I can't tell what are the commands to use but is there any chance that the user which is running elasticsearch process can not write files on /tmp/snapshot_production?

Like: can you run something like the following on both machines?

sudo -u elasticsearch touch /tmp/snapshot_production/foo.txt
sudo -u elasticsearch rm /tmp/snapshot_production/foo.txt
7

Hi Dadoonet,

I found the exact error.

Error: Elasticsearch userid & groupid is different in both servers

So, now i have chaged the userid & groupid of elasticsearch in "server2". To keep it same in both server.

But after changing it , the elasticsearch service is not starting in server2.

Please guide me in solving this problem.

Thanks,
Gogul.

8

Logs?

9

Hi Dadoonet,

I found the issue...

Once i changed the permission in /etc/elasticsearch

service stared.

Now it is working fine. Thanks a lot for guiding me to solve this issue.

Thanks,
Gogul.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.