ES INSISTS on trying to create repos in wrong place

Elastic Stack Elasticsearch
1

Setup:

  • Azure
  • Ubuntu 16.04
  • Elasticsearch 1.7.3 (yes I know it's old, but we're not ready to change it yet)

Challenge: We implemented backups. At some point, the path.repo didn't match the place the backup script was trying to create the repo. That was back in April.

It seems that from then until now, the system has kept trying to create those repos, despite restart of Elastic, and a complete system reboot. The Task Queue is empty.

Log sample:

[2017-09-20 06:15:02,295][WARN ][repositories ] [Dev Client ES Node 1] failed to create repository [fs][2017-04-25] org.elasticsearch.common.inject.CreationException: Guice creation errors:

Error injecting constructor, org.elasticsearch.repositories.RepositoryException: [2017-04-25] location [/media/backups2/es/2017-04-25] doesn't match any of the locations specified by path.repo at org.elasticsearch.repositories.fs.FsRepository.(Unknown Source) while locating org.elasticsearch.repositories.fs.FsRepository while locating org.elasticsearch.repositories.Repository

HOW do I get rid of this? It's gotten to the point where this is bringing Elasticsearch down.

Thanks in advance!

2

Elasticsearch 1.7.3

Well. At least you should really upgrade to latest 1.7 IMHO.

You probably need to add in elasticsearch.yml on every node:

path.repo: /media/backups2/es/2017-04-25
3

Hi @dadoonet That date is the name of the repo. The correct path is /media/backups4/es/

Briefly (several days), the script tried /media/backups2/es/. The script and path.repo are now correctly both on backups4. The problem is that it's still trying to create the ones that were incorrect, 5 months later.

What I'm getting at here is, WHERE are these tries coming from? What engine or file or whatever is it that is hanging on to these things?

Yes, I could point something to the wrong path so that it could let go, but I want to know in case this happens again on a production server. There, I won't have the luxury of being able to add a new path to path.repo and restarting ES.

I hope I've clarified this.

4

I believe there is a repository in the cluster state which is referencing it.
Would be good I think to create a fake dir, start one Es node. Delete the repository.
Remove that fake dir.
Restart the node. Then the other nodes.

My 2 cents

5

I assume there's no API to change or delete that reference?

What command would I need to use to check that the reference is indeed in the Cluster State?

6

I'm unsure it's there in:

https://www.elastic.co/guide/en/elasticsearch/reference/1.7/cluster-state.html

May be list the repositories with:

GET /_snapshot
7

That's it done, thanks! I unfortunately did have to restart the Node, but adding that path to path.repo did allow me to remove the offending items. Thanks again!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.