Setting path.data stops the EC2 stuff

Steinar_Bang · December 24, 2013, 12:35pm

Platform: ES 0.9.7 on Ubuntu 12.04 on AWS EC2 instance

I now have the EC2 clustering stuff working on itself.

What I'm trying to do now, is index on ephemeral storage (where I have
space) and save the index to s3, and restore it on EC2 instance startup.

Note: I am only running a single node in a single EC2 instance.

Note2: I will try the EBS stuff also but right now I have an s3 bucket
set up for me, and can experiment with it. I need someone with EC2 admin
privileges to set up the EBS volume (...or so I think...?).

My problem now is that if I set data.path to in
/etc/elasticsearch/elasticsearch.yml to /mnt/elasticsearch (to get
ephemeral storage, since my main problem here is running out of disk
space on an EC2 instance that is started and stopped) then the EC2
discovery and S3 settings seem to be skipped.

Here is the complete elasticsearch.yml file with keys XXX'd:
https://gist.github.com/steinarb/8094353

Ie. if I don't set path.data then the file
/var/log/elasticsearch/mysecretname-cluster.log is written to, and messages
like this, is output:
[2013-12-24 12:17:14,341][TRACE][discovery.ec2 ] [Foster, Bill] building dynamic unicast discovery nodes...
[2013-12-24 12:17:14,342][DEBUG][discovery.ec2 ] [Foster, Bill] using dynamic discovery nodes []
...
[2013-12-24 12:17:17,715][DEBUG][gateway.s3 ] [Foster, Bill] reading state from gateway org.elasticsearch.gateway.shared.SharedStorageGateway$1@2eb9f428 ...

If I do set data.path, /var/log/elasticsearch/elasticsearch.log is
written to and no .ec2 and .s3 messages can be found in the log.

(Once I get this working, one more problem will be to ensure that
/mnt/elasticsearch is created and owned by user elasticsearch before ES
is started.)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/878uvaig05.fsf%40dod.no.
For more options, visit https://groups.google.com/groups/opt_out.

Steinar_Bang · December 27, 2013, 10:58pm

Steinar Bang sb@dod.no:

My problem now is that if I set data.path to in
/etc/elasticsearch/elasticsearch.yml to /mnt/elasticsearch (to get
ephemeral storage, since my main problem here is running out of disk
space on an EC2 instance that is started and stopped) then the EC2
discovery and S3 settings seem to be skipped.

Here is the complete elasticsearch.yml file with keys XXX'd:
/etc/elasticsearch/elasticsearch.yml from deb package with EC2 stuff added, used in Ubuntu 12.04 EC2 instance · GitHub

I figured out what the problem was: I had a leading space in the line
with the path.data setting, ie.
path.data=/mnt/elasticsearch
instead of
path.data=/mnt/elasticsearch

Once I removed the space, things started working.

Ie. they startet working, if /mnt/elasticsearch existed and if it
was writable by elasticsearch.

So I need to these commands when the EC2 instance wakes up, and before
the ES server is started:
mkdir -p /mnt/elasticsearch
chown elasticsearch.elasticsearch /mnt/elasticsearch

Is there an easy way to do this from the /etc/init.d/elasticsearch
script, before the process user is changed from root to elasticsearch, I
wonder...?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/87wqipewa1.fsf%40dod.no.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · December 28, 2013, 8:22am

Ha! Nice you found the issue.

About your last question, are you still using S3?
As Karel wrote, you should really use EBS instead of S3 gateway.
I guess that in that case with EBS your drives will be automatically mounted when your instance wake up.

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 27 déc. 2013 à 23:58, Steinar Bang sb@dod.no a écrit :

Steinar Bang sb@dod.no:

My problem now is that if I set data.path to in
/etc/elasticsearch/elasticsearch.yml to /mnt/elasticsearch (to get
ephemeral storage, since my main problem here is running out of disk
space on an EC2 instance that is started and stopped) then the EC2
discovery and S3 settings seem to be skipped.

Here is the complete elasticsearch.yml file with keys XXX'd:
/etc/elasticsearch/elasticsearch.yml from deb package with EC2 stuff added, used in Ubuntu 12.04 EC2 instance · GitHub

I figured out what the problem was: I had a leading space in the line
with the path.data setting, ie.
path.data=/mnt/elasticsearch
instead of
path.data=/mnt/elasticsearch

Once I removed the space, things started working.

Ie. they startet working, if /mnt/elasticsearch existed and if it
was writable by elasticsearch.

So I need to these commands when the EC2 instance wakes up, and before
the ES server is started:
mkdir -p /mnt/elasticsearch
chown elasticsearch.elasticsearch /mnt/elasticsearch

Is there an easy way to do this from the /etc/init.d/elasticsearch
script, before the process user is changed from root to elasticsearch, I
wonder...?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/87wqipewa1.fsf%40dod.no.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/333F63BD-7D10-405B-B08F-ADCA1BB0C981%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.

Steinar_Bang · December 29, 2013, 10:56pm

David Pilato david@pilato.fr:

Ha! Nice you found the issue.
About your last question, are you still using S3?

Well, I completed the experiment to see what happened. Turns out the S3
operations generated a bit of unexpected cost (many I/O operations
during indexing).

Also a restore to /mnt/elasticsearch after EC2 instance start took 10-15
minutes for a 25GB index.

As Karel wrote, you should really use EBS instead of S3 gateway.
I guess that in that case with EBS your drives will be automatically
mounted when your instance wake up.

Yep. Got an EBS drive in operation now. Will test it tomorrow.

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/87ob3ze06d.fsf%40dod.no.
For more options, visit https://groups.google.com/groups/opt_out.

Steinar_Bang · December 29, 2013, 11:01pm

Steinar Bang sb@dod.no:

Ie. they startet working, if /mnt/elasticsearch existed and if it
was writable by elasticsearch.

So I need to these commands when the EC2 instance wakes up, and before
the ES server is started:
mkdir -p /mnt/elasticsearch
chown elasticsearch.elasticsearch /mnt/elasticsearch

Is there an easy way to do this from the /etc/init.d/elasticsearch
script, before the process user is changed from root to elasticsearch, I
wonder...?

What I did to get things working automatically, was to add these lines
to the /etc/rc.local file:

Create a volatile disk directory for elastiscsearch to work on

The ES index will be recreated from the last checkpoint in S3

mkdir -p /mnt/elasticsearch
chown elasticsearch.elasticsearch /mnt/elasticsearch/

This made ES start correctly, and start restoring the saved index when
started during the boot. ES was up with a restored 25.5GB index in
around 10-15 minutes after the EC2 instance boot.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/87k3endzy6.fsf%40dod.no.
For more options, visit https://groups.google.com/groups/opt_out.

Steinar_Bang · December 30, 2013, 1:58pm

I now have an EBS device mounted at /data on my EC2, Ubuntu 12.04
instance.

My plan was, to:

Create /data/elasticsearch
Make /data/elasticsearch owned by
Set path.data to /data/elasticsearch
Reboot the EC2 instance, letting ES start and fill up
/data/elasticsearch with stuff stored on S3
Remove the S3 settings from /etc/elasticsearch/elasticsearch.yml
Wait until the index doesn't change anymore
Reboot the EC2 instance

Unfortunately ES came up again, but without any indices. Should this
have worked? Or will it fail, because if things I don't understand about
how gateways work?

Thanks!

Steinar

Here are the log messages from the restart been removed:
[2013-12-30 13:06:35,434][INFO ][node [2013-12-30 13:06:35,436][INFO ][node [2013-12-30 13:06:35,585][INFO ][plugins [2013-12-30 13:06:38,812][DEBUG][discovery.zen [2013-12-30 13:06:38,817][DEBUG][discovery.zen [2013-12-30 13:06:38,819][DEBUG][discovery.ec2 [2013-12-30 13:06:38,820][DEBUG][discovery.zen.elect [2013-12-30 13:06:38,822][DEBUG][discovery.zen.fd [2013-12-30 13:06:38,827][DEBUG][discovery.zen.fd [2013-12-30 13:06:40,008][DEBUG][discovery.ec2 [2013-12-30 13:06:41,503][DEBUG][gateway.local [2013-12-30 13:06:41,904][DEBUG][gateway.local.state.meta [2013-12-30 13:06:41,906][DEBUG][gateway.local.state.meta [2013-12-30 13:06:41,907][DEBUG][gateway.local.state.meta [2013-12-30 13:06:41,907][DEBUG][gateway.local.state.meta [2013-12-30 13:06:41,916][DEBUG][gateway.local [2013-12-30 13:06:41,944][INFO ][node [2013-12-30 13:06:41,945][INFO ][node [2013-12-30 13:06:42,147][INFO ][transport [2013-12-30 13:06:42,168][TRACE][discovery [2013-12-30 13:06:43,304][TRACE][discovery.ec2 [2013-12-30 13:06:43,304][DEBUG][discovery.ec2 [2013-12-30 13:06:44,979][TRACE][discovery.ec2 [2013-12-30 13:06:44,979][DEBUG][discovery.ec2 [2013-12-30 13:06:46,632][TRACE][discovery.ec2 [2013-12-30 13:06:46,632][DEBUG][discovery.ec2 [2013-12-30 13:06:46,633][TRACE][discovery.ec2 [2013-12-30 13:06:46,633][DEBUG][discovery.ec2 [2013-12-30 13:06:46,641][INFO ][cluster.service [2013-12-30 13:06:46,690][TRACE][discovery [2013-12-30 13:06:46,690][INFO ][discovery [2013-12-30 13:06:46,698][DEBUG][gateway.local.state.meta [2013-12-30 13:06:46,699][DEBUG][gateway.local.state.meta [2013-12-30 13:06:46,750][INFO ][http [2013-12-30 13:06:46,750][INFO ][node [2013-12-30 13:06:46,871][INFO ][gateway after the S3 settings had
] [Tefral the Surveyor] version[0.90.7], pid[800], build[36897d0/2013-11-13T12:06:54Z]
] [Tefral the Surveyor] initializing ...
] [Tefral the Surveyor] loaded [river-datagen, cloud-aws], sites [head]
.ping.multicast] [Tefral the Surveyor] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
.ping.unicast] [Tefral the Surveyor] using initial hosts [], with concurrent_connects [10]
] [Tefral the Surveyor] using ping.timeout [3s], master_election.filter_client [true], master_election.filter_data [false]
] [Tefral the Surveyor] using minimum_master_nodes [-1]
] [Tefral the Surveyor] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
] [Tefral the Surveyor] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
] [Tefral the Surveyor] using host_type [PRIVATE_IP], tags [{}], groups [[]] with any_group [true], availability_zones [[]]
] [Tefral the Surveyor] using initial_shards [quorum], list_timeout [30s]
] [Tefral the Surveyor] using gateway.local.auto_import_dangled [YES], with gateway.local.dangling_timeout [2h]
] [Tefral the Surveyor] [_river] failed to find metadata for existing index location
] [Tefral the Surveyor] [my_index] failed to find metadata for existing index location
] [Tefral the Surveyor] took 3ms to load state
.state.shards] [Tefral the Surveyor] took 9ms to load started shards state
] [Tefral the Surveyor] initialized
] [Tefral the Surveyor] starting ...
] [Tefral the Surveyor] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/172.31.31.127:9300]}
] [Tefral the Surveyor] waiting for 30s for the initial state to be set by the discovery
] [Tefral the Surveyor] building dynamic unicast discovery nodes...
] [Tefral the Surveyor] using dynamic discovery nodes []
] [Tefral the Surveyor] building dynamic unicast discovery nodes...
] [Tefral the Surveyor] using dynamic discovery nodes []
] [Tefral the Surveyor] building dynamic unicast discovery nodes...
] [Tefral the Surveyor] using dynamic discovery nodes []
] [Tefral the Surveyor] full ping responses: {none}
] [Tefral the Surveyor] filtered ping responses: (filter_client[true], filter_data[false]) {none}
] [Tefral the Surveyor] new_master [Tefral the Surveyor][rOtRMCncTPy7-jnjO1B_5g][inet[/172.31.31.127:9300]], reason: zen-disco-join (elected_as_master)
] [Tefral the Surveyor] initial state set from discovery
] [Tefral the Surveyor] csisearch-cluster/rOtRMCncTPy7-jnjO1B_5g
] [Tefral the Surveyor] [_river] failed to find metadata for existing index location
] [Tefral the Surveyor] [my_index] failed to find metadata for existing index location
] [Tefral the Surveyor] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/172.31.31.127:9200]}
] [Tefral the Surveyor] started
] [Tefral the Surveyor] recovered [0] indices into cluster_state

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/87d2kee906.fsf_-_%40dod.no.
For more options, visit https://groups.google.com/groups/opt_out.

Steinar_Bang · December 30, 2013, 5:24pm

Steinar Bang sb@dod.no:
[snip! Attempted transitioning from S3 to local gateway on EBS]
Unfortunately ES came up again, but without any indices. Should this
have worked? Or will it fail, because if things I don't understand
about how gateways work?

I googled a bit, and came up with the gateway.local.auto_import_dangled
setting, which I tried setting to yes, and restarting ES. But still no
luck, it finds the two indexes, but it's unable to load them since it
doesn't find the metadata:

[2013-12-30 17:19:08,305][INFO ][node ] [Tefral the Surveyor] stopping ...
[2013-12-30 17:19:08,384][INFO ][node ] [Tefral the Surveyor] stopped
[2013-12-30 17:19:08,385][INFO ][node ] [Tefral the Surveyor] closing ...
[2013-12-30 17:19:08,401][INFO ][node ] [Tefral the Surveyor] closed
[2013-12-30 17:19:10,714][INFO ][node ] [Senor Muerte] version[0.90.7], pid[1699], build[36897d0/2013-11-13T12:06:54Z]
[2013-12-30 17:19:10,714][INFO ][node ] [Senor Muerte] initializing ...
[2013-12-30 17:19:10,738][INFO ][plugins ] [Senor Muerte] loaded [river-datagen, cloud-aws], sites [head]
[2013-12-30 17:19:13,211][DEBUG][discovery.zen.ping.multicast] [Senor Muerte] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
[2013-12-30 17:19:13,216][DEBUG][discovery.zen.ping.unicast] [Senor Muerte] using initial hosts , with concurrent_connects [10]
[2013-12-30 17:19:13,218][DEBUG][discovery.ec2 ] [Senor Muerte] using ping.timeout [3s], master_election.filter_client [true], master_election.filter_data [false]
[2013-12-30 17:19:13,219][DEBUG][discovery.zen.elect ] [Senor Muerte] using minimum_master_nodes [-1]
[2013-12-30 17:19:13,220][DEBUG][discovery.zen.fd ] [Senor Muerte] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-12-30 17:19:13,226][DEBUG][discovery.zen.fd ] [Senor Muerte] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-12-30 17:19:13,838][DEBUG][discovery.ec2 ] [Senor Muerte] using host_type [PRIVATE_IP], tags [{}], groups [] with any_group [true], availability_zones []
[2013-12-30 17:19:15,177][DEBUG][gateway.local ] [Senor Muerte] using initial_shards [quorum], list_timeout [30s]
[2013-12-30 17:19:15,563][DEBUG][gateway.local.state.meta ] [Senor Muerte] using gateway.local.auto_import_dangled [YES], with gateway.local.dangling_timeout [2h]
[2013-12-30 17:19:15,578][DEBUG][gateway.local.state.meta ] [Senor Muerte] [_river] failed to find metadata for existing index location
[2013-12-30 17:19:15,578][DEBUG][gateway.local.state.meta ] [Senor Muerte] [my_index] failed to find metadata for existing index location
[2013-12-30 17:19:15,578][DEBUG][gateway.local.state.meta ] [Senor Muerte] took 14ms to load state
[2013-12-30 17:19:15,580][DEBUG][gateway.local.state.shards] [Senor Muerte] took 1ms to load started shards state
[2013-12-30 17:19:15,606][INFO ][node ] [Senor Muerte] initialized
[2013-12-30 17:19:15,606][INFO ][node ] [Senor Muerte] starting ...
[2013-12-30 17:19:15,751][INFO ][transport ] [Senor Muerte] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/172.31.31.127:9300]}
[2013-12-30 17:19:15,757][TRACE][discovery ] [Senor Muerte] waiting for 30s for the initial state to be set by the discovery
[2013-12-30 17:19:16,661][TRACE][discovery.ec2 ] [Senor Muerte] building dynamic unicast discovery nodes...
[2013-12-30 17:19:16,661][DEBUG][discovery.ec2 ] [Senor Muerte] using dynamic discovery nodes
[2013-12-30 17:19:18,334][TRACE][discovery.ec2 ] [Senor Muerte] building dynamic unicast discovery nodes...
[2013-12-30 17:19:18,334][DEBUG][discovery.ec2 ] [Senor Muerte] using dynamic discovery nodes
[2013-12-30 17:19:20,002][TRACE][discovery.ec2 ] [Senor Muerte] building dynamic unicast discovery nodes...
[2013-12-30 17:19:20,002][DEBUG][discovery.ec2 ] [Senor Muerte] using dynamic discovery nodes
[2013-12-30 17:19:20,003][TRACE][discovery.ec2 ] [Senor Muerte] full ping responses: {none}
[2013-12-30 17:19:20,003][DEBUG][discovery.ec2 ] [Senor Muerte] filtered ping responses: (filter_client[true], filter_data[false]) {none}
[2013-12-30 17:19:20,011][INFO ][cluster.service ] [Senor Muerte] new_master [Senor Muerte][66JvcflfSBKhkS-eUoJNzg][inet[/172.31.31.127:9300]], reason: zen-disco-join (elected_as_master)
[2013-12-30 17:19:20,058][TRACE][discovery ] [Senor Muerte] initial state set from discovery
[2013-12-30 17:19:20,058][INFO ][discovery ] [Senor Muerte] csisearch-cluster/66JvcflfSBKhkS-eUoJNzg
[2013-12-30 17:19:20,066][DEBUG][gateway.local.state.meta ] [Senor Muerte] [_river] failed to find metadata for existing index location
[2013-12-30 17:19:20,067][DEBUG][gateway.local.state.meta ] [Senor Muerte] [my_index] failed to find metadata for existing index location
[2013-12-30 17:19:20,093][INFO ][http ] [Senor Muerte] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/172.31.31.127:9200]}
[2013-12-30 17:19:20,094][INFO ][node ] [Senor Muerte] started
[2013-12-30 17:19:20,112][INFO ][gateway ] [Senor Muerte] recovered [0] indices into cluster_state

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/878uv2dzfr.fsf%40dod.no.
For more options, visit https://groups.google.com/groups/opt_out.

Steinar_Bang · December 30, 2013, 5:41pm

Is this a way that could be used, perhaps?
http://tech.superhappykittymeow.com/?p=296

Ie.

Put the S3 stuff back into /etc/elasticsearch/elasticsearch.yml
Restart ES, and hopefully it will find the stuff it needs in S3 and
come up with the two indexes
Use the curl/sed commands in the above URL to extract metadata from
port 9200
Remove the S3 stuff from /etc/elasticsearch/elasticsearch.yml
Restart ES, and it again will be unable to load the indexes, because
of the missing metadata
Use curl to PUT the saved metadata into port 9200

I think I will try this. Worst case is that I scrap the indexes on
disk, and build up new indexes from S3.

(And an even worse case is that I will have to refeed the data, but even
that is no tragedy)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/874n5qdyoi.fsf%40dod.no.
For more options, visit https://groups.google.com/groups/opt_out.

Steinar_Bang · December 30, 2013, 6:30pm

Steinar Bang sb@dod.no:

Is this a way that could be used, perhaps?
http://tech.superhappykittymeow.com/?p=296

I tried, but no luck, I'm afraid.

Ie.

Put the S3 stuff back into /etc/elasticsearch/elasticsearch.yml

Restart ES, and hopefully it will find the stuff it needs in S3 and
come up with the two indexes

Use the curl/sed commands in the above URL to extract metadata from
port 9200

Remove the S3 stuff from /etc/elasticsearch/elasticsearch.yml

Restart ES, and it again will be unable to load the indexes, because
of the missing metadata

Use curl to PUT the saved metadata into port 9200

At this stage I had the metadata for the indexes in place, but no
documents in either index.

Then I took a restart of ES, and it came up with the two indexes empty
still, and the old data om /data/elasticsearch removed.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/87zjnichtg.fsf%40dod.no.
For more options, visit https://groups.google.com/groups/opt_out.

Steinar_Bang · December 30, 2013, 6:41pm

One thing I could try, is snapshot to volatile storage, scrap the index
and restore from volatile storage.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html

Is that viable, with a 25.5GB index on an EC2 instance?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/87vby6chbw.fsf%40dod.no.
For more options, visit https://groups.google.com/groups/opt_out.

Steinar_Bang · December 30, 2013, 7:16pm

Steinar Bang sb@dod.no:

One thing I could try, is snapshot to volatile storage, scrap the index
and restore from volatile storage.
Elasticsearch Platform — Find real-time answers at scale | Elastic

Is that viable, with a 25.5GB index on an EC2 instance?

Trying to set up the backup repository according to the above document,
fails:
ubuntu@ip-172-31-31-127:~$ curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{

"type": "fs",
"settings": {
    "location": "/mnt/backups/my_backup",
    "compress": false
}
}'
No handler found for uri [/_snapshot/my_backup] and method [PUT]
ubuntu@ip-172-31-31-127:~$

Do I need to enable the backup/restore module in some way?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/87r48ucfp0.fsf%40dod.no.
For more options, visit https://groups.google.com/groups/opt_out.

Steinar_Bang · December 30, 2013, 7:22pm

Steinar Bang sb@dod.no:

Steinar Bang sb@dod.no:
One thing I could try, is snapshot to volatile storage, scrap the index
and restore from volatile storage.
Elasticsearch Platform — Find real-time answers at scale | Elastic

Trying to set up the backup repository according to the above document,
fails:
[snip!]
Do I need to enable the backup/restore module in some way?

Nope, looks like I would need 1.0.0-beta2, rather than the 0.9.7 I have
currently installed:
Elasticsearch Platform — Find real-time answers at scale | Elastic

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/87mwjicfeh.fsf%40dod.no.
For more options, visit https://groups.google.com/groups/opt_out.

Steinar_Bang · December 30, 2013, 8:55pm

Steinar Bang sb@dod.no:

Nope, looks like I would need 1.0.0-beta2, rather than the 0.9.7 I have
currently installed:
Elasticsearch Platform — Find real-time answers at scale | Elastic

I'm now up and running on EBS storage on ES 1.0.0-beta2, after having
first created a snapshot with volatile storage+S3 gateway, then switched
to EBS storage and no S3 gateway, and restored the snapshot.

The restored index seems to be identical to the original index, with all
of the tests I've been able to apply to it.

Thanks for the snapshot/restore facility! Good work!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/87iou6cb40.fsf%40dod.no.
For more options, visit https://groups.google.com/groups/opt_out.