Correct, this would create new indices.
On Apr 21, 2014, at 2:50 PM, Mohit Anchlia mohitanchlia@gmail.com wrote:
If I understand this correct with this pattern evey restore is a new index?
curl -XPOST "localhost:9200/snapshot/my_backup/snapshot_1/restore" -d '{
"indices": "index_1,index_2",
"ignore_unavailable": "true",
"include_global_state": false,
"rename_pattern": "index(.+)",
"rename_replacement": "restored_index$1"
}'On Mon, Apr 21, 2014 at 2:32 PM, Andrew Selden andrew.selden@elasticsearch.com wrote:
Hi,How many indices you create is really up to you. Typically you would make a snapshot of an index once and then schedule incremental snapshots at regular intervals.
Andrew
On Apr 21, 2014, at 2:24 PM, Mohit Anchlia mohitanchlia@gmail.com wrote:
Thanks for the information. To keep the indexes in near real time sync I'll end up creating thousands of indexes every day alone with this method, correct?
On Mon, Apr 21, 2014 at 1:37 PM, Andrew Selden andrew.selden@elasticsearch.com wrote:
Hi,I can see how the documentation may not be as clear as it should be on this point. Let me try to explain. If you refer back to [1], you will see an example restore command:
$ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -d '{"indices": "index_1,index_2", "ignore_unavailable": "true", "include_global_state": false, "rename_pattern": "index_(.+)", "rename_replacement": "restored_index_$1"}'
The important parts to notice are: rename_pattern and rename_replacement. What this means is that, for any index in the repository named with the prefix 'index_', it will be restored to the cluster under the new name 'restored_index_$1', where $1 is the name of the original index. So "index_1" gets restored to "restored_index_1", meanwhile you can keep index_1 open and servicing requests.The basic operational workflow is that you would perform the restore operation with a rename. Once the restore is complete, do some sanity checking to confirm that you have in fact restored what you intended to. Finally, to make "restored_index_1" live and take "index_1" offline without impacting production service, you would use an atomic rename alias command [2].
This assumes you have been using aliases to manage your indices. For example, typically you would set up an alias to point to a 'logical' index to mask one or more 'physical' indices. This is highly recommended as it gives you the flexibility to do things like hot-swap restored/live indices as in the above example.
Hope that helps.
Andrew
- Elasticsearch Platform — Find real-time answers at scale | Elastic
- Elasticsearch Platform — Find real-time answers at scale | Elastic
On Apr 21, 2014, at 12:36 PM, Mohit Anchlia mohitanchlia@gmail.com wrote:
This document seem to suggest that to do the restore elasticsearch need to close the index, which means index is not available during that time while restore is occurring?
On Mon, Apr 21, 2014 at 12:31 PM, Andrew Selden andrew.selden@elasticsearch.com wrote:
Hi,Snapshot/restore is fine to run on an active, production system. The indices are still available for normal indexing and searching operations. You can throttle the snapshot operation if you are concerned about impacting production quality of service [1].
I would suggest avoiding using scroll as a backup strategy. Keeping a scroll open for a long time can be taxing on an actively used production cluster. You are much better off using the snapshot/restore functionality.
Andrew
On Apr 21, 2014, at 9:23 AM, Mohit Anchlia mohitanchlia@gmail.com wrote:
Our data centers are a actively used and need to be highly available. Does snapshot and restore happen in the background and indexes are still available to the clients for read/write?
Can Scroll be used to read indexes continuously and insert/update them?
On Mon, Apr 21, 2014 at 5:57 AM, Alexander Reelsen alr@spinscale.de wrote:
hey,snapshot and restore might be an option, if you are willing to accept a lag a few minutes behind. another option might be to index/update/delete all your data into two different clusters, so you dont need replication at all?
--Alex
On Fri, Apr 18, 2014 at 10:19 PM, Mohit Anchlia mohitanchlia@gmail.com wrote:
As I understand there is currently no feature that does async replication between 2 clusters or even within the same cluster, but we have a need to write one. What would be the best way to do it in elasticsearch? I was thinking of leveraging Scroll for this.--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrUdDdiqy62yHaaS6bJJ08_txDCNNXR8rGr%3DRGY8gAv-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9G8539fUmrzLWgkXoSYAEyUYkjUbyjeBiNVhBdarWTFA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpqVdoSWv552vh3ZOL7siBiMUCGRUV5NcBwAFoWPisVLQ%40mail.gmail.com.For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/B852C0E8-A2BE-4C23-B14A-EE2593A4D7ED%40elasticsearch.com.For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWr14cNv05GAjoxtXVtUY4-uPzaxt200tp_spDNXvO_tyg%40mail.gmail.com.For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/C9E4589A-FBCB-4FC3-9492-35B8A722EB20%40elasticsearch.com.For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9AC0A79F-B695-4F7D-80FC-D39C9FD2F15E%40elasticsearch.com.For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpGTUH3cqrsqeGTvPucQpRTY%3D0%2BHYB_YRQ8tFqJQ1myCQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8EEA285B-9B46-4DDC-A9C9-B82134AA0D8A%40elasticsearch.com.
For more options, visit https://groups.google.com/d/optout.