Issue help - snapshot creation failure

Hi all,
I've been trying to create the snapshot for the ES system. I have added the path.repo and restarted the services and it all looks fine.

When I tried to create the snapshot like following

curl -XPUT 'http://10.85.7.55:9200/_snapshot/backup' -d '{
"type":"fs",
"settings":{
"location":"/var/log/elasticsearch/backup",
"compress":true
}
}'

I got the errors like

{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[backup] [wDj7ox5RSf6XiAhd4LEUgQ, 'RemoteTransportException[[dataNode1][10.8                                           fy]]; nested: RepositoryMissingException[[backup] missing];'], [PD8oAfGZQuiO_zyVxuduKw, 'RemoteTransportException[[dataNode2][10.85.7.54:9300][internal:ad                                           MissingException[[backup] missing];']]"}],"type":"repository_verification_exception","reason":"[backup] [wDj7ox5RSf6XiAhd4LEUgQ, 'RemoteTransportException                                           n/repository/verify]]; nested: RepositoryMissingException[[backup] missing];'], [PD8oAfGZQuiO_zyVxuduKw, 'RemoteTransportException[[dataNode2][10.85.7.54:                                           ested: RepositoryMissingException[[backup] missing];']]"},"status":500}[root@she01irisexp03 elasticsearch]

Can anybody advise how I can get past of this hurdle? Thanks a lot.

Anyone can advise what has caused this issue please?

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

I think you did not share the same dir on node 10.85.7.54.

Thank you so much David. Just reformatted my codes.
Sorry I am still new to Elasticsearch world. How do I share the dir on node 10.85.7.54?

To do snapshot/restore on a Shared FS, you need to have a Shared FS.

Copying docs here:

The shared file system repository ("type": "fs") uses the shared file system to store snapshots. In order to register the shared file system repository it is necessary to mount the same shared filesystem to the same location on all master and data nodes. This location (or one of its parent directories) must be registered in the path.repo setting on all master and data nodes.

Forgive me if I am being ignorant here as I am really new to this technology ...

I am not sure if I am doing the right thing to do snapshot on the shared FS. Here below is the partition info for the system. Do I have to register my path.repo to /dev/shm or anywhere else? Right now I registered my path.repo under /var/log/elasticsearch/backup.

[root@she01irisexp03 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_she01irisexp03-lv_root
                       50G  3.9G   43G   9% /
tmpfs                  63G     0   63G   0% /dev/shm
/dev/sda1             477M   42M  411M  10% /boot
/dev/mapper/vg_she01irisexp03-lv_home
                      1.1T  906M  992G   1% /home
/dev/mapper/vg--store-data
                      3.6T  1.4T  2.1T  41% /data

Thanks a lot.

You have more than one node in your cluster, right?

GET _cat/nodes?v

You need to do the same thing on all nodes

Yes, I have 2 other nodes seemingly.

host       ip         heap.percent ram.percent load node.role master name
10.85.7.55 10.85.7.55           21          84 0.12 d         *      master
10.85.7.54 10.85.7.54           30          91 0.34 d         -      dataNode2
10.85.7.53 10.85.7.53           19          88 0.23 d         -      dataNode1

If I need to do the same thing on all nodes, do I need to register path.repo for all nodes? How do I do that by pointing to the right node? Did I do the right thing for path.repo registration? How do I include all nodes in?

Really appreciated your patience. It's so helpful.

As the doc says you have to mount the same remote dir on each node and add it to elasticsearch.yml on all nodes.

Thanks a lot David. That's actually something confused me.

I was trying to mount the dir to each node, like following

path.repo: ["/var/log/elasticsearch/backup", "/var/log/elasticsearch/backup1", "/var/log/elasticsearch/backup2"]

And I'm sure they are all accessible and have been granted the full permit (chmod 777). All looks good after I restarted the elasticsearch service.

I did the restart as following

curl -XPUT 'http://10.85.7.55:9200/_cluster/settings' -d '{"transient":{"cluster.routing.allocation.enable":"none"}}'
service elasticsearch restart
curl -XPUT 'http://10.85.7.55:9200/_cluster/settings' -d '{"transient":{"cluster.routing.allocation.enable":"all"}}'

I now suspect if I have to use the data1/data2 nodes IPs 10.85.7.54/53 instead of the 10.85.7.55 as above shown to restart the services on data1 and data2? And how do I mount the dir to data1 and data2? Do the directories have to be on 54/53 servers locally?

Please advise. Thanks a lot.

Do the directories have to be on 54/53 servers locally?

Yes

Then how do I specify the server name/IP and location in Elasticsearch.yml file? As I mentioned, it is now

path.repo: ["/var/log/elasticsearch/backup", "/var/log/elasticsearch/backup1", "/var/log/elasticsearch/backup2"]

The first dir is for master node, and the remaining two dirs are for datanode1 and 2. How do I specify if the dir is on data node1/2? Are they automatically pointed to the data node 1/2 respectively if I put the dir here, say, /var/log/elasticsearch/backup1 will be pointed to data node 1 and /var/log/elasticsearch/backup2 will be pointed to data node 2 respectively? So what I only need to do is to mount the dir /var/log/elasticsearch/backup1 on data node 1 and dir /var/log/elasticsearch/backup2 on data node 2?

Sorry I know it's basic but truly I really appreciate the advice.

Mount your shared FS in /mnt/elasticsearch on every node.

Then just add this single path in path.repo

Just done what you advised.

Have the new entry input in elasticsearch.yml file

path.repo: ["/mnt/elasticsearch"]

restarted all nodes and elasticsearch services and then run

curl -XPUT 'http://10.85.7.55:9200/_snapshot/backup' -d '{
"type":"fs",
"settings":{
"location":"/mnt/elasticsearch",
"compress":true
}
}'

The errors got persistent

{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[backup] [PD8oAfGZQuiO_zyVxuduKw, 'RemoteTransportException[[dataNode2][10.85.7.54:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[backup] missing];'], [wDj7ox5RSf6XiAhd4LEUgQ, 'RemoteTransportException[[dataNode1][10.85.7.53:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[backup] missing];']]"}],"type":"repository_verification_exception","reason":"[backup] [PD8oAfGZQuiO_zyVxuduKw, 'RemoteTransportException[[dataNode2][10.85.7.54:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[backup] missing];'], [wDj7ox5RSf6XiAhd4LEUgQ, 'RemoteTransportException[[dataNode1][10.85.7.53:9300][internal:admin/repository/verify]]; nested: RepositoryMissingException[[backup] missing];']]"},"status":500}[root@she01irisexp03 elasticsearch]

I am wondering if I have to change the IP 10.85.7.55 in my command line as I have to do the snapshot for all the nodes? But if I do the localhost as the alternative, this URL can't even be connected

curl: (7) couldn't connect to host

Do I still miss something? Thanks.

Can you share the full logs from the restart of the 3 nodes?

# LICENSE EXPIRED ON [Sunday, December 11, 2016]. IF YOU HAVE A NEW LICENSE, PLEASE
# UPDATE IT. OTHERWISE, PLEASE REACH OUT TO YOUR SUPPORT CONTACT.
# 
# COMMERCIAL PLUGINS OPERATING WITH REDUCED FUNCTIONALITY
# - marvel
#  - The agent will stop collecting cluster and indices metrics
#  - The agent will stop automatically cleaning indices older than [marvel.history.duration]
[2017-04-04 18:06:38,408][INFO ][cluster.routing.allocation.decider] [master] updating [cluster.routing.allocation.enable] from [ALL] to [NONE]
[2017-04-04 18:06:47,094][INFO ][node                     ] [master] stopping ...
[2017-04-04 18:06:47,520][INFO ][node                     ] [master] stopped
[2017-04-04 18:06:47,520][INFO ][node                     ] [master] closing ...
[2017-04-04 18:06:47,526][INFO ][node                     ] [master] closed
[2017-04-04 18:06:48,705][WARN ][bootstrap                ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2017-04-04 18:06:48,832][INFO ][node                     ] [master] version[2.4.0], pid[172565], build[ce9f0c7/2016-08-29T09:14:17Z]
[2017-04-04 18:06:48,832][INFO ][node                     ] [master] initializing ...
[2017-04-04 18:06:49,252][INFO ][plugins                  ] [master] modules [reindex, lang-expression, lang-groovy], plugins [license, marvel-agent], sites []
[2017-04-04 18:06:49,270][INFO ][env                      ] [master] using [1] data paths, mounts [[/data (/dev/mapper/vg--store-data)]], net usable_space [2tb], net total_space [3.5tb], spins? [possibly], types [ext4]
[2017-04-04 18:06:49,270][INFO ][env                      ] [master] heap size [30.7gb], compressed ordinary object pointers [true]
[2017-04-04 18:06:49,270][WARN ][env                      ] [master] max file descriptors [65535] for elasticsearch process likely too low, consider increasing to at least [65536]
[2017-04-04 18:06:51,305][INFO ][node                     ] [master] initialized
[2017-04-04 18:06:51,305][INFO ][node                     ] [master] starting ...
[2017-04-04 18:06:51,553][INFO ][transport                ] [master] publish_address {10.85.7.55:9300}, bound_addresses {10.85.7.55:9300}
[2017-04-04 18:06:51,598][INFO ][discovery                ] [master] VzB_ELS_Cluster/KA5-VTNJQCCmf1QzLtsjsw
[2017-04-04 18:06:54,513][DEBUG][action.admin.indices.create] [master] no known master node, scheduling a retry
[2017-04-04 18:06:54,513][DEBUG][action.admin.indices.create] [master] no known master node, scheduling a retry
[2017-04-04 18:06:54,627][INFO ][cluster.service          ] [master] new_master {master}{KA5-VTNJQCCmf1QzLtsjsw}{10.85.7.55}{10.85.7.55:9300}{slave=false, master=true}, added {{dataNode2}{PD8oAfGZQuiO_zyVxuduKw}{10.85.7.54}{10.85.7.54:9300}{slave=true, master=false},{dataNode1}{wDj7ox5RSf6XiAhd4LEUgQ}{10.85.7.53}{10.85.7.53:9300}{slave=true, master=false},}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-04-04 18:06:54,906][INFO ][http                     ] [master] publish_address {10.85.7.55:9200}, bound_addresses {10.85.7.55:9200}
[2017-04-04 18:06:54,906][INFO ][node                     ] [master] started
[2017-04-04 18:06:55,232][WARN ][repositories.fs          ] [master] The specified location [/var/log/elasticsearch/backup] doesn't start with any repository paths specified by the path.repo setting: [[/mnt/elasticsearch]] 
[2017-04-04 18:06:55,233][WARN ][repositories             ] [master] failed to create repository [fs][my_backup]
org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, RepositoryException[[my_backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
  at org.elasticsearch.repositories.fs.FsRepository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.fs.FsRepository
  while locating org.elasticsearch.repositories.Repository

1 error
	at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:360)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
	at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
	at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
	at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
	at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
	at org.elasticsearch.repositories.RepositoriesService.clusterChanged(RepositoriesService.java:299)
	at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
	at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: RepositoryException[[my_backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
	at org.elasticsearch.repositories.fs.FsRepository.<init>(FsRepository.java:81)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:50)
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
	at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:54)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:201)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:879)
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
	... 12 more
[2017-04-04 18:06:55,237][WARN ][repositories             ] [master] failed to create repository [my_backup]
RepositoryException[[my_backup] failed to create repository]; nested: CreationException[Guice creation errors:

1) Error injecting constructor, RepositoryException[[my_backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
  at org.elasticsearch.repositories.fs.FsRepository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.fs.FsRepository
  while locating org.elasticsearch.repositories.Repository

1 error]; nested: RepositoryException[[my_backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo];
	at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:411)
	at org.elasticsearch.repositories.RepositoriesService.clusterChanged(RepositoriesService.java:299)
	at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
	at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, RepositoryException[[my_backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
  at org.elasticsearch.repositories.fs.FsRepository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.fs.FsRepository
  while locating org.elasticsearch.repositories.Repository

1 error
	at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:360)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
	at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
	at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
	at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
	at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
	... 8 more
Caused by: RepositoryException[[my_backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
	at org.elasticsearch.repositories.fs.FsRepository.<init>(FsRepository.java:81)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:50)
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
	at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:54)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:201)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:879)
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
	... 12 more
[2017-04-04 18:06:55,238][WARN ][repositories.fs          ] [master] The specified location [/var/log/elasticsearch/backup] doesn't start with any repository paths specified by the path.repo setting: [[/mnt/elasticsearch]] 
[2017-04-04 18:06:55,239][WARN ][repositories             ] [master] failed to create repository [fs][backup]
org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, RepositoryException[[backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
  at org.elasticsearch.repositories.fs.FsRepository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.fs.FsRepository
  while locating org.elasticsearch.repositories.Repository

1 error
	at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:360)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
	at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
	at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
	at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
	at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
	at org.elasticsearch.repositories.RepositoriesService.clusterChanged(RepositoriesService.java:299)
	at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
	at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: RepositoryException[[backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
	at org.elasticsearch.repositories.fs.FsRepository.<init>(FsRepository.java:81)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:50)
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
	at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:54)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:201)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:879)
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
	... 12 more
[2017-04-04 18:06:55,239][WARN ][repositories             ] [master] failed to create repository [backup]
RepositoryException[[backup] failed to create repository]; nested: CreationException[Guice creation errors:

1) Error injecting constructor, RepositoryException[[backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
  at org.elasticsearch.repositories.fs.FsRepository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.fs.FsRepository
  while locating org.elasticsearch.repositories.Repository

1 error]; nested: RepositoryException[[backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo];
	at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:411)
	at org.elasticsearch.repositories.RepositoriesService.clusterChanged(RepositoriesService.java:299)
	at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
	at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, RepositoryException[[backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
  at org.elasticsearch.repositories.fs.FsRepository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.fs.FsRepository
  while locating org.elasticsearch.repositories.Repository
1 error
	at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:360)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
	at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
	at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
	at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
	at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
	... 8 more
Caused by: RepositoryException[[backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
	at org.elasticsearch.repositories.fs.FsRepository.<init>(FsRepository.java:81)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:50)
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
	at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:54)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:201)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:879)
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
	... 12 more
[2017-04-04 18:06:55,374][INFO ][license.plugin.core      ] [master] license [77b34686-c643-41ad-9beb-903badd0e73e] - expired
[2017-04-04 18:06:55,378][ERROR][license.plugin.core      ] [master] 
#
# LICENSE EXPIRED ON [Sunday, December 11, 2016]. IF YOU HAVE A NEW LICENSE, PLEASE
# UPDATE IT. OTHERWISE, PLEASE REACH OUT TO YOUR SUPPORT CONTACT.
# 
# COMMERCIAL PLUGINS OPERATING WITH REDUCED FUNCTIONALITY
# - marvel
#  - The agent will stop collecting cluster and indices metrics
#  - The agent will stop automatically cleaning indices older than [marvel.history.duration]
[2017-04-04 18:06:55,552][INFO ][gateway                  ] [master] recovered [128] indices into cluster_state
[2017-04-04 18:07:27,944][INFO ][cluster.routing.allocation] [master] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[fios-explorer_2017-04-04_f1d9d5b4-fccf-4232-80f0-e6a194a11088][0]] ...]).
[2017-04-04 18:07:58,783][INFO ][repositories             ] [master] update repository [backup]
[2017-04-04 18:07:58,794][WARN ][repositories.fs          ] [master] The specified location [/var/log/elasticsearch/backup] doesn't start with any repository paths specified by the path.repo setting: [[/mnt/elasticsearch]] 
[2017-04-04 18:07:58,794][WARN ][repositories             ] [master] failed to create repository [fs][my_backup]
org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, RepositoryException[[my_backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
  at org.elasticsearch.repositories.fs.FsRepository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.fs.FsRepository
  while locating org.elasticsearch.repositories.Repository

1 error
	at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:360)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
	at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
	at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
	at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
	at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
	at org.elasticsearch.repositories.RepositoriesService.clusterChanged(RepositoriesService.java:299)
	at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
	at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: RepositoryException[[my_backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
	at org.elasticsearch.repositories.fs.FsRepository.<init>(FsRepository.java:81)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:50)
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
	at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:54)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:201)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:879)
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
	... 12 more
[2017-04-04 18:07:58,795][WARN ][repositories             ] [master] failed to create repository [my_backup]
RepositoryException[[my_backup] failed to create repository]; nested: CreationException[Guice creation errors:

1) Error injecting constructor, RepositoryException[[my_backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
  at org.elasticsearch.repositories.fs.FsRepository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.fs.FsRepository
  while locating org.elasticsearch.repositories.Repository

1 error]; nested: RepositoryException[[my_backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo];
	at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:411)
	at org.elasticsearch.repositories.RepositoriesService.clusterChanged(RepositoriesService.java:299)
	at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
	at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, RepositoryException[[my_backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
  at org.elasticsearch.repositories.fs.FsRepository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.fs.FsRepository
  while locating org.elasticsearch.repositories.Repository

1 error
	at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:360)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
	at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
	at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
	at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
	at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
	... 8 more
Caused by: RepositoryException[[my_backup] location [/var/log/elasticsearch/backup] doesn't match any of the locations specified by path.repo]
	at org.elasticsearch.repositories.fs.FsRepository.<init>(FsRepository.java:81)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:50)
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
	at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:54)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:886)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:201)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:879)
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
	... 12 more