# Setting path.data stops the EC2 stuff

**URL:** <https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010>\
**Category:** Elasticsearch\
**Created:** [December 24, 2013, 12:35pm UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010 "2013-12-24T12:35:06Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Steinar\_Bang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steinar_bang/32/122285_2.png) [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\
**Post date:** [December 24, 2013, 12:35pm UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010/1 "2013-12-24T12:35:06Z")

</div>

Platform: ES 0.9.7 on Ubuntu 12.04 on AWS EC2 instance

I now have the EC2 clustering stuff working on itself.

What I'm trying to do now, is index on ephemeral storage (where I have  
space) and save the index to s3, and restore it on EC2 instance startup.

Note: I am only running a single node in a single EC2 instance.

Note2: I will try the EBS stuff also but right now I have an s3 bucket  
set up for me, and can experiment with it. I need someone with EC2 admin  
privileges to set up the EBS volume (...or so I think...?).

My problem now is that if I set data.path to in  
/etc/elasticsearch/elasticsearch.yml to /mnt/elasticsearch (to get  
ephemeral storage, since my main problem here is running out of disk  
space on an EC2 instance that is started and stopped) then the EC2  
discovery and S3 settings seem to be skipped.

Here is the complete elasticsearch.yml file with keys XXX'd:  
[https://gist.github.com/steinarb/8094353](https://gist.github.com/steinarb/8094353)

Ie. if I don't set path.data then the file  
/var/log/elasticsearch/mysecretname-cluster.log is written to, and messages  
like this, is output:  
[2013-12-24 12:17:14,341][TRACE][discovery.ec2] [Foster, Bill] building dynamic unicast discovery nodes...  
[2013-12-24 12:17:14,342][DEBUG][discovery.ec2] [Foster, Bill] using dynamic discovery nodes []  
...  
[2013-12-24 12:17:17,715][DEBUG][gateway.s3] [Foster, Bill] reading state from gateway org.elasticsearch.gateway.shared.SharedStorageGateway$1@2eb9f428 ...

If I do set data.path, /var/log/elasticsearch/elasticsearch.log is  
written to and no .ec2 and .s3 messages can be found in the log.

(Once I get this working, one more problem will be to ensure that  
/mnt/elasticsearch is created and owned by user elasticsearch before ES  
is started.)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/878uvaig05.fsf%40dod.no](https://groups.google.com/d/msgid/elasticsearch/878uvaig05.fsf%40dod.no).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Steinar\_Bang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steinar_bang/32/122285_2.png) [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\
**Post date:** [December 27, 2013, 10:58pm UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010/2 "2013-12-27T22:58:30Z")

</div>

> > > > > Steinar Bang [sb@dod.no](mailto:sb@dod.no):

> My problem now is that if I set data.path to in  
> /etc/elasticsearch/elasticsearch.yml to /mnt/elasticsearch (to get  
> ephemeral storage, since my main problem here is running out of disk  
> space on an EC2 instance that is started and stopped) then the EC2  
> discovery and S3 settings seem to be skipped.

> Here is the complete elasticsearch.yml file with keys XXX'd:  
> [/etc/elasticsearch/elasticsearch.yml from deb package with EC2 stuff added, used in Ubuntu 12.04 EC2 instance · GitHub](https://gist.github.com/steinarb/8094353)

I figured out what the problem was: I had a leading space in the line  
with the path.data setting, ie.  
path.data=/mnt/elasticsearch  
instead of  
path.data=/mnt/elasticsearch

Once I removed the space, things started working.

Ie. they startet working, _if_ /mnt/elasticsearch existed and _if_ it  
was writable by elasticsearch.

So I need to these commands when the EC2 instance wakes up, and before  
the ES server is started:  
mkdir -p /mnt/elasticsearch  
chown elasticsearch.elasticsearch /mnt/elasticsearch

Is there an easy way to do this from the /etc/init.d/elasticsearch  
script, before the process user is changed from root to elasticsearch, I  
wonder...?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/87wqipewa1.fsf%40dod.no](https://groups.google.com/d/msgid/elasticsearch/87wqipewa1.fsf%40dod.no).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 28, 2013, 8:22am UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010/3 "2013-12-28T08:22:47Z")

</div>

Ha! Nice you found the issue.

About your last question, are you still using S3?  
As Karel wrote, you should really use EBS instead of S3 gateway.  
I guess that in that case with EBS your drives will be automatically mounted when your instance wake up.

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 27 déc. 2013 à 23:58, Steinar Bang [sb@dod.no](mailto:sb@dod.no) a écrit :

> > > > > Steinar Bang [sb@dod.no](mailto:sb@dod.no):

> My problem now is that if I set data.path to in  
> /etc/elasticsearch/elasticsearch.yml to /mnt/elasticsearch (to get  
> ephemeral storage, since my main problem here is running out of disk  
> space on an EC2 instance that is started and stopped) then the EC2  
> discovery and S3 settings seem to be skipped.

> Here is the complete elasticsearch.yml file with keys XXX'd:  
> [/etc/elasticsearch/elasticsearch.yml from deb package with EC2 stuff added, used in Ubuntu 12.04 EC2 instance · GitHub](https://gist.github.com/steinarb/8094353)

I figured out what the problem was: I had a leading space in the line  
with the path.data setting, ie.  
path.data=/mnt/elasticsearch  
instead of  
path.data=/mnt/elasticsearch

Once I removed the space, things started working.

Ie. they startet working, _if_ /mnt/elasticsearch existed and _if_ it  
was writable by elasticsearch.

So I need to these commands when the EC2 instance wakes up, and before  
the ES server is started:  
mkdir -p /mnt/elasticsearch  
chown elasticsearch.elasticsearch /mnt/elasticsearch

Is there an easy way to do this from the /etc/init.d/elasticsearch  
script, before the process user is changed from root to elasticsearch, I  
wonder...?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/87wqipewa1.fsf%40dod.no](https://groups.google.com/d/msgid/elasticsearch/87wqipewa1.fsf%40dod.no).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/333F63BD-7D10-405B-B08F-ADCA1BB0C981%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/333F63BD-7D10-405B-B08F-ADCA1BB0C981%40pilato.fr).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Steinar\_Bang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steinar_bang/32/122285_2.png) [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\
**Post date:** [December 29, 2013, 10:56pm UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010/4 "2013-12-29T22:56:26Z")

</div>

> > > > > David Pilato [david@pilato.fr](mailto:david@pilato.fr):

> Ha! Nice you found the issue.  
> About your last question, are you still using S3?

Well, I completed the experiment to see what happened. Turns out the S3  
operations generated a bit of unexpected cost (many I/O operations  
during indexing).

Also a restore to /mnt/elasticsearch after EC2 instance start took 10-15  
minutes for a 25GB index.

> As Karel wrote, you should really use EBS instead of S3 gateway.  
> I guess that in that case with EBS your drives will be automatically  
> mounted when your instance wake up.

Yep. Got an EBS drive in operation now. Will test it tomorrow.

Thanks!

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/87ob3ze06d.fsf%40dod.no](https://groups.google.com/d/msgid/elasticsearch/87ob3ze06d.fsf%40dod.no).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Steinar\_Bang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steinar_bang/32/122285_2.png) [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\
**Post date:** [December 29, 2013, 11:01pm UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010/5 "2013-12-29T23:01:21Z")

</div>

> > > > > Steinar Bang [sb@dod.no](mailto:sb@dod.no):

> Ie. they startet working, _if_ /mnt/elasticsearch existed and _if_ it  
> was writable by elasticsearch.

> So I need to these commands when the EC2 instance wakes up, and before  
> the ES server is started:  
> mkdir -p /mnt/elasticsearch  
> chown elasticsearch.elasticsearch /mnt/elasticsearch

> Is there an easy way to do this from the /etc/init.d/elasticsearch  
> script, before the process user is changed from root to elasticsearch, I  
> wonder...?

What I did to get things working automatically, was to add these lines  
to the /etc/rc.local file:

# Create a volatile disk directory for elastiscsearch to work on

# The ES index will be recreated from the last checkpoint in S3

mkdir -p /mnt/elasticsearch  
chown elasticsearch.elasticsearch /mnt/elasticsearch/

This made ES start correctly, and start restoring the saved index when  
started during the boot. ES was up with a restored 25.5GB index in  
around 10-15 minutes after the EC2 instance boot.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/87k3endzy6.fsf%40dod.no](https://groups.google.com/d/msgid/elasticsearch/87k3endzy6.fsf%40dod.no).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Steinar\_Bang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steinar_bang/32/122285_2.png) [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\
**Post date:** [December 30, 2013, 1:58pm UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010/6 "2013-12-30T13:58:01Z")

</div>

I now have an EBS device mounted at /data on my EC2, Ubuntu 12.04  
instance.

My plan was, to:

1. Create /data/elasticsearch
2. Make /data/elasticsearch owned by
3. Set path.data to /data/elasticsearch
4. Reboot the EC2 instance, letting ES start and fill up  
/data/elasticsearch with stuff stored on S3
5. Remove the S3 settings from /etc/elasticsearch/elasticsearch.yml
6. Wait until the index doesn't change anymore
7. Reboot the EC2 instance

Unfortunately ES came up again, but without any indices. Should this  
have worked? Or will it fail, because if things I don't understand about  
how gateways work?

Thanks!

- Steinar

Here are the log messages from the restart after the S3 settings had  
been removed:  
[2013-12-30 13:06:35,434][INFO][node] [Tefral the Surveyor] version[0.90.7], pid[800], build[36897d0/2013-11-13T12:06:54Z]  
[2013-12-30 13:06:35,436][INFO][node] [Tefral the Surveyor] initializing ...  
[2013-12-30 13:06:35,585][INFO][plugins] [Tefral the Surveyor] loaded [river-datagen, cloud-aws], sites [head]  
[2013-12-30 13:06:38,812][DEBUG][discovery.zen.ping.multicast] [Tefral the Surveyor] using group [224.2.2.4], with port [54328], ttl [3], and address [null]  
[2013-12-30 13:06:38,817][DEBUG][discovery.zen.ping.unicast] [Tefral the Surveyor] using initial hosts [], with concurrent\_connects [10]  
[2013-12-30 13:06:38,819][DEBUG][discovery.ec2] [Tefral the Surveyor] using ping.timeout [3s], master\_election.filter\_client [true], master\_election.filter\_data [false]  
[2013-12-30 13:06:38,820][DEBUG][discovery.zen.elect] [Tefral the Surveyor] using minimum\_master\_nodes [-1]  
[2013-12-30 13:06:38,822][DEBUG][discovery.zen.fd] [Tefral the Surveyor] [master] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]  
[2013-12-30 13:06:38,827][DEBUG][discovery.zen.fd] [Tefral the Surveyor] [node] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]  
[2013-12-30 13:06:40,008][DEBUG][discovery.ec2] [Tefral the Surveyor] using host\_type [PRIVATE\_IP], tags [{}], groups [[]] with any\_group [true], availability\_zones [[]]  
[2013-12-30 13:06:41,503][DEBUG][gateway.local] [Tefral the Surveyor] using initial\_shards [quorum], list\_timeout [30s]  
[2013-12-30 13:06:41,904][DEBUG][gateway.local.state.meta] [Tefral the Surveyor] using gateway.local.auto\_import\_dangled [YES], with gateway.local.dangling\_timeout [2h]  
[2013-12-30 13:06:41,906][DEBUG][gateway.local.state.meta] [Tefral the Surveyor] [\_river] failed to find metadata for existing index location  
[2013-12-30 13:06:41,907][DEBUG][gateway.local.state.meta] [Tefral the Surveyor] [my\_index] failed to find metadata for existing index location  
[2013-12-30 13:06:41,907][DEBUG][gateway.local.state.meta] [Tefral the Surveyor] took 3ms to load state  
[2013-12-30 13:06:41,916][DEBUG][gateway.local.state.shards] [Tefral the Surveyor] took 9ms to load started shards state  
[2013-12-30 13:06:41,944][INFO][node] [Tefral the Surveyor] initialized  
[2013-12-30 13:06:41,945][INFO][node] [Tefral the Surveyor] starting ...  
[2013-12-30 13:06:42,147][INFO][transport] [Tefral the Surveyor] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/172.31.31.127:9300]}  
[2013-12-30 13:06:42,168][TRACE][discovery] [Tefral the Surveyor] waiting for 30s for the initial state to be set by the discovery  
[2013-12-30 13:06:43,304][TRACE][discovery.ec2] [Tefral the Surveyor] building dynamic unicast discovery nodes...  
[2013-12-30 13:06:43,304][DEBUG][discovery.ec2] [Tefral the Surveyor] using dynamic discovery nodes []  
[2013-12-30 13:06:44,979][TRACE][discovery.ec2] [Tefral the Surveyor] building dynamic unicast discovery nodes...  
[2013-12-30 13:06:44,979][DEBUG][discovery.ec2] [Tefral the Surveyor] using dynamic discovery nodes []  
[2013-12-30 13:06:46,632][TRACE][discovery.ec2] [Tefral the Surveyor] building dynamic unicast discovery nodes...  
[2013-12-30 13:06:46,632][DEBUG][discovery.ec2] [Tefral the Surveyor] using dynamic discovery nodes []  
[2013-12-30 13:06:46,633][TRACE][discovery.ec2] [Tefral the Surveyor] full ping responses: {none}  
[2013-12-30 13:06:46,633][DEBUG][discovery.ec2] [Tefral the Surveyor] filtered ping responses: (filter\_client[true], filter\_data[false]) {none}  
[2013-12-30 13:06:46,641][INFO][cluster.service] [Tefral the Surveyor] new\_master [Tefral the Surveyor][rOtRMCncTPy7-jnjO1B\_5g][inet[/172.31.31.127:9300]], reason: zen-disco-join (elected\_as\_master)  
[2013-12-30 13:06:46,690][TRACE][discovery] [Tefral the Surveyor] initial state set from discovery  
[2013-12-30 13:06:46,690][INFO][discovery] [Tefral the Surveyor] csisearch-cluster/rOtRMCncTPy7-jnjO1B\_5g  
[2013-12-30 13:06:46,698][DEBUG][gateway.local.state.meta] [Tefral the Surveyor] [\_river] failed to find metadata for existing index location  
[2013-12-30 13:06:46,699][DEBUG][gateway.local.state.meta] [Tefral the Surveyor] [my\_index] failed to find metadata for existing index location  
[2013-12-30 13:06:46,750][INFO][http] [Tefral the Surveyor] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/172.31.31.127:9200]}  
[2013-12-30 13:06:46,750][INFO][node] [Tefral the Surveyor] started  
[2013-12-30 13:06:46,871][INFO][gateway] [Tefral the Surveyor] recovered [0] indices into cluster\_state

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/87d2kee906.fsf\_-\_%40dod.no](https://groups.google.com/d/msgid/elasticsearch/87d2kee906.fsf_-_%40dod.no).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Steinar\_Bang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steinar_bang/32/122285_2.png) [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\
**Post date:** [December 30, 2013, 5:24pm UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010/7 "2013-12-30T17:24:40Z")

</div>

> > > > > Steinar Bang [sb@dod.no](mailto:sb@dod.no):  
> > > > > [snip! Attempted transitioning from S3 to local gateway on EBS]  
> > > > > Unfortunately ES came up again, but without any indices. Should this  
> > > > > have worked? Or will it fail, because if things I don't understand  
> > > > > about how gateways work?

I googled a bit, and came up with the gateway.local.auto\_import\_dangled  
setting, which I tried setting to yes, and restarting ES. But still no  
luck, it finds the two indexes, but it's unable to load them since it  
doesn't find the metadata:

[2013-12-30 17:19:08,305][INFO][node] [Tefral the Surveyor] stopping ...  
[2013-12-30 17:19:08,384][INFO][node] [Tefral the Surveyor] stopped  
[2013-12-30 17:19:08,385][INFO][node] [Tefral the Surveyor] closing ...  
[2013-12-30 17:19:08,401][INFO][node] [Tefral the Surveyor] closed  
[2013-12-30 17:19:10,714][INFO][node] [Senor Muerte] version[0.90.7], pid[1699], build[36897d0/2013-11-13T12:06:54Z]  
[2013-12-30 17:19:10,714][INFO][node] [Senor Muerte] initializing ...  
[2013-12-30 17:19:10,738][INFO][plugins] [Senor Muerte] loaded [river-datagen, cloud-aws], sites [head]  
[2013-12-30 17:19:13,211][DEBUG][discovery.zen.ping.multicast] [Senor Muerte] using group [224.2.2.4], with port [54328], ttl [3], and address [null]  
[2013-12-30 17:19:13,216][DEBUG][discovery.zen.ping.unicast] [Senor Muerte] using initial hosts , with concurrent\_connects [10]  
[2013-12-30 17:19:13,218][DEBUG][discovery.ec2] [Senor Muerte] using ping.timeout [3s], master\_election.filter\_client [true], master\_election.filter\_data [false]  
[2013-12-30 17:19:13,219][DEBUG][discovery.zen.elect] [Senor Muerte] using minimum\_master\_nodes [-1]  
[2013-12-30 17:19:13,220][DEBUG][discovery.zen.fd] [Senor Muerte] [master] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]  
[2013-12-30 17:19:13,226][DEBUG][discovery.zen.fd] [Senor Muerte] [node] uses ping\_interval [1s], ping\_timeout [30s], ping\_retries [3]  
[2013-12-30 17:19:13,838][DEBUG][discovery.ec2] [Senor Muerte] using host\_type [PRIVATE\_IP], tags [{}], groups [] with any\_group [true], availability\_zones []  
[2013-12-30 17:19:15,177][DEBUG][gateway.local] [Senor Muerte] using initial\_shards [quorum], list\_timeout [30s]  
[2013-12-30 17:19:15,563][DEBUG][gateway.local.state.meta] [Senor Muerte] using gateway.local.auto\_import\_dangled [YES], with gateway.local.dangling\_timeout [2h]  
[2013-12-30 17:19:15,578][DEBUG][gateway.local.state.meta] [Senor Muerte] [\_river] failed to find metadata for existing index location  
[2013-12-30 17:19:15,578][DEBUG][gateway.local.state.meta] [Senor Muerte] [my\_index] failed to find metadata for existing index location  
[2013-12-30 17:19:15,578][DEBUG][gateway.local.state.meta] [Senor Muerte] took 14ms to load state  
[2013-12-30 17:19:15,580][DEBUG][gateway.local.state.shards] [Senor Muerte] took 1ms to load started shards state  
[2013-12-30 17:19:15,606][INFO][node] [Senor Muerte] initialized  
[2013-12-30 17:19:15,606][INFO][node] [Senor Muerte] starting ...  
[2013-12-30 17:19:15,751][INFO][transport] [Senor Muerte] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/172.31.31.127:9300]}  
[2013-12-30 17:19:15,757][TRACE][discovery] [Senor Muerte] waiting for 30s for the initial state to be set by the discovery  
[2013-12-30 17:19:16,661][TRACE][discovery.ec2] [Senor Muerte] building dynamic unicast discovery nodes...  
[2013-12-30 17:19:16,661][DEBUG][discovery.ec2] [Senor Muerte] using dynamic discovery nodes   
[2013-12-30 17:19:18,334][TRACE][discovery.ec2] [Senor Muerte] building dynamic unicast discovery nodes...  
[2013-12-30 17:19:18,334][DEBUG][discovery.ec2] [Senor Muerte] using dynamic discovery nodes   
[2013-12-30 17:19:20,002][TRACE][discovery.ec2] [Senor Muerte] building dynamic unicast discovery nodes...  
[2013-12-30 17:19:20,002][DEBUG][discovery.ec2] [Senor Muerte] using dynamic discovery nodes   
[2013-12-30 17:19:20,003][TRACE][discovery.ec2] [Senor Muerte] full ping responses: {none}  
[2013-12-30 17:19:20,003][DEBUG][discovery.ec2] [Senor Muerte] filtered ping responses: (filter\_client[true], filter\_data[false]) {none}  
[2013-12-30 17:19:20,011][INFO][cluster.service] [Senor Muerte] new\_master [Senor Muerte][66JvcflfSBKhkS-eUoJNzg][inet[/172.31.31.127:9300]], reason: zen-disco-join (elected\_as\_master)  
[2013-12-30 17:19:20,058][TRACE][discovery] [Senor Muerte] initial state set from discovery  
[2013-12-30 17:19:20,058][INFO][discovery] [Senor Muerte] csisearch-cluster/66JvcflfSBKhkS-eUoJNzg  
[2013-12-30 17:19:20,066][DEBUG][gateway.local.state.meta] [Senor Muerte] [\_river] failed to find metadata for existing index location  
[2013-12-30 17:19:20,067][DEBUG][gateway.local.state.meta] [Senor Muerte] [my\_index] failed to find metadata for existing index location  
[2013-12-30 17:19:20,093][INFO][http] [Senor Muerte] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/172.31.31.127:9200]}  
[2013-12-30 17:19:20,094][INFO][node] [Senor Muerte] started  
[2013-12-30 17:19:20,112][INFO][gateway] [Senor Muerte] recovered [0] indices into cluster\_state

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/878uv2dzfr.fsf%40dod.no](https://groups.google.com/d/msgid/elasticsearch/878uv2dzfr.fsf%40dod.no).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Steinar\_Bang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steinar_bang/32/122285_2.png) [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\
**Post date:** [December 30, 2013, 5:41pm UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010/8 "2013-12-30T17:41:01Z")

</div>

Is this a way that could be used, perhaps?  
[http://tech.superhappykittymeow.com/?p=296](http://tech.superhappykittymeow.com/?p=296)

Ie.

1. Put the S3 stuff back into /etc/elasticsearch/elasticsearch.yml
2. Restart ES, and hopefully it will find the stuff it needs in S3 and  
come up with the two indexes
3. Use the curl/sed commands in the above URL to extract metadata from  
port 9200
4. Remove the S3 stuff from /etc/elasticsearch/elasticsearch.yml
5. Restart ES, and it again will be unable to load the indexes, because  
of the missing metadata
6. Use curl to PUT the saved metadata into port 9200

I think I will try this. Worst case is that I scrap the indexes on  
disk, and build up new indexes from S3.

(And an even worse case is that I will have to refeed the data, but even  
that is no tragedy)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/874n5qdyoi.fsf%40dod.no](https://groups.google.com/d/msgid/elasticsearch/874n5qdyoi.fsf%40dod.no).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Steinar\_Bang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steinar_bang/32/122285_2.png) [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\
**Post date:** [December 30, 2013, 6:30pm UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010/9 "2013-12-30T18:30:35Z")

</div>

> > > > > Steinar Bang [sb@dod.no](mailto:sb@dod.no):

> Is this a way that could be used, perhaps?  
> [http://tech.superhappykittymeow.com/?p=296](http://tech.superhappykittymeow.com/?p=296)

I tried, but no luck, I'm afraid.

> Ie.
> 
> 1. Put the S3 stuff back into /etc/elasticsearch/elasticsearch.yml
> 2. Restart ES, and hopefully it will find the stuff it needs in S3 and  
> come up with the two indexes
> 3. Use the curl/sed commands in the above URL to extract metadata from  
> port 9200
> 4. Remove the S3 stuff from /etc/elasticsearch/elasticsearch.yml
> 5. Restart ES, and it again will be unable to load the indexes, because  
> of the missing metadata
> 6. Use curl to PUT the saved metadata into port 9200

At this stage I had the metadata for the indexes in place, but no  
documents in either index.

Then I took a restart of ES, and it came up with the two indexes empty  
still, and the old data om /data/elasticsearch removed.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/87zjnichtg.fsf%40dod.no](https://groups.google.com/d/msgid/elasticsearch/87zjnichtg.fsf%40dod.no).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Steinar\_Bang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steinar_bang/32/122285_2.png) [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\
**Post date:** [December 30, 2013, 6:41pm UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010/10 "2013-12-30T18:41:07Z")

</div>

One thing I could try, is snapshot to volatile storage, scrap the index  
and restore from volatile storage.  
[http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html)

Is that viable, with a 25.5GB index on an EC2 instance?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/87vby6chbw.fsf%40dod.no](https://groups.google.com/d/msgid/elasticsearch/87vby6chbw.fsf%40dod.no).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Steinar\_Bang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steinar_bang/32/122285_2.png) [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\
**Post date:** [December 30, 2013, 7:16pm UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010/11 "2013-12-30T19:16:27Z")

</div>

> > > > > Steinar Bang [sb@dod.no](mailto:sb@dod.no):

> One thing I could try, is snapshot to volatile storage, scrap the index  
> and restore from volatile storage.  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html)

> Is that viable, with a 25.5GB index on an EC2 instance?

Trying to set up the backup repository according to the above document,  
fails:  
ubuntu@ip-172-31-31-127:~$ curl -XPUT '[http://localhost:9200/\_snapshot/my\_backup](http://localhost:9200/_snapshot/my_backup)' -d '{

> ```
> "type": "fs",
> "settings": {
> "location": "/mnt/backups/my_backup",
> "compress": false
> }
> 
> ```
> 
> }'  
> No handler found for uri [/\_snapshot/my\_backup] and method [PUT]  
> ubuntu@ip-172-31-31-127:~$

Do I need to enable the backup/restore module in some way?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/87r48ucfp0.fsf%40dod.no](https://groups.google.com/d/msgid/elasticsearch/87r48ucfp0.fsf%40dod.no).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Steinar\_Bang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steinar_bang/32/122285_2.png) [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\
**Post date:** [December 30, 2013, 7:22pm UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010/12 "2013-12-30T19:22:46Z")

</div>

> > > > > Steinar Bang [sb@dod.no](mailto:sb@dod.no):

> > > > > Steinar Bang [sb@dod.no](mailto:sb@dod.no):  
> > > > > One thing I could try, is snapshot to volatile storage, scrap the index  
> > > > > and restore from volatile storage.  
> > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html)

> Trying to set up the backup repository according to the above document,  
> fails:  
> [snip!]  
> Do I need to enable the backup/restore module in some way?

Nope, looks like I would need 1.0.0-beta2, rather than the 0.9.7 I have  
currently installed:  
[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/1-0-0-beta2-released/)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/87mwjicfeh.fsf%40dod.no](https://groups.google.com/d/msgid/elasticsearch/87mwjicfeh.fsf%40dod.no).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Steinar\_Bang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steinar_bang/32/122285_2.png) [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\
**Post date:** [December 30, 2013, 8:55pm UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010/13 "2013-12-30T20:55:27Z")

</div>

> > > > > Steinar Bang [sb@dod.no](mailto:sb@dod.no):

> Nope, looks like I would need 1.0.0-beta2, rather than the 0.9.7 I have  
> currently installed:  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/1-0-0-beta2-released/)

I'm now up and running on EBS storage on ES 1.0.0-beta2, after having  
first created a snapshot with volatile storage+S3 gateway, then switched  
to EBS storage and no S3 gateway, and restored the snapshot.

The restored index seems to be identical to the original index, with all  
of the tests I've been able to apply to it.

Thanks for the snapshot/restore facility! Good work! 🙂

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/87iou6cb40.fsf%40dod.no](https://groups.google.com/d/msgid/elasticsearch/87iou6cb40.fsf%40dod.no).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:58am UTC](https://discuss.elastic.co/t/setting-path-data-stops-the-ec2-stuff/15010/14 "2017-07-06T01:58:58Z")

</div>