# Manually attaching an index **URL:** \ **Category:** Elasticsearch\ **Created:** [August 22, 2011, 4:57am UTC](https://discuss.elastic.co/t/manually-attaching-an-index/5216 "2011-08-22T04:57:53Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![dottom](https://avatars.discourse-cdn.com/v4/letter/d/c68b51/32.png) [@dottom](https://discuss.elastic.co/u/dottom)\ **Post date:** [August 22, 2011, 4:57am UTC](https://discuss.elastic.co/t/manually-attaching-an-index/5216/1 "2011-08-22T04:57:53Z")
If a full disk backup was made of an index directory (with ES stopped), is there a way to manually attach or import it? A related question is if there is a way to merge two indexes with the same name, from two separate nodes/clusters? I could write a custom script to read data from cluster 1 and index onto cluster 2, but would be nice if there was a way to simply copy a few TB of data and "import" or "attach" it. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 22, 2011, 6:43pm UTC](https://discuss.elastic.co/t/manually-attaching-an-index/5216/2 "2011-08-22T18:43:05Z")
There isn't a way to "import" a specific index, since the fact that its "there" in the cluster is also stored in the metadata state of the cluster. I agree, it would be nice to have some sort of notion of import / merge, but, tricky to get the API right... (in a distributed env, i.e., where is the data you plan to import from?) On Mon, Aug 22, 2011 at 7:57 AM, Tom Le [dottom@gmail.com](mailto:dottom@gmail.com) wrote: > If a full disk backup was made of an index directory (with ES > stopped), is there a way to manually attach or import it? > > A related question is if there is a way to merge two indexes with the > same name, from two separate nodes/clusters? I could write a custom > script to read data from cluster 1 and index onto cluster 2, but would > be nice if there was a way to simply copy a few TB of data and > "import" or "attach" it. ---
**Author:** ![dottom](https://avatars.discourse-cdn.com/v4/letter/d/c68b51/32.png) [@dottom](https://discuss.elastic.co/u/dottom)\ **Post date:** [August 22, 2011, 9:55pm UTC](https://discuss.elastic.co/t/manually-attaching-an-index/5216/3 "2011-08-22T21:55:09Z")
We are looking at more traditional IT scenarios, such as performing a disk backup using SAN shadow copy (shut down ES, create snapshot, start ES - we can copy hundreds of TB's very quickly). We also want to move a lot of data from a single standalone node to another for testing purposes. For example, I have a 10-TB of indexed data that I maintain as the control data set, and I want multiple researchers to have their own independent standalone node for testing. Each researcher needs a different index, where each index is 1-TB in size. With traditional db's, we'd export and import. For ES, I have to maintain 10 separate instances of 1-TB each so i can create a large tarball and ship it to a researcher. I'd like to be able to "export" just specific indexes (making sure mappings and all other settings are the same). There was also a scenario where a client was pointing to the wrong ES cluster and we indexed 300-GB of data (using a unique index name, so all the data resided in one index). In this case, I did not have access to the raw data and would have to write a script to query the cluster, and then re-index to the correct cluster. An "export" feature here where I could move an entire index by simply copying files would be nice. On Mon, Aug 22, 2011 at 11:43 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > There isn't a way to "import" a specific index, since the fact that its > "there" in the cluster is also stored in the metadata state of the cluster. > I agree, it would be nice to have some sort of notion of import / merge, > but, tricky to get the API right... (in a distributed env, i.e., where is > the data you plan to import from?) > > On Mon, Aug 22, 2011 at 7:57 AM, Tom Le [dottom@gmail.com](mailto:dottom@gmail.com) wrote: > > > If a full disk backup was made of an index directory (with ES > > stopped), is there a way to manually attach or import it? > > > > A related question is if there is a way to merge two indexes with the > > same name, from two separate nodes/clusters? I could write a custom > > script to read data from cluster 1 and index onto cluster 2, but would > > be nice if there was a way to simply copy a few TB of data and > > "import" or "attach" it. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 29, 2011, 6:31pm UTC](https://discuss.elastic.co/t/manually-attaching-an-index/5216/4 "2011-08-29T18:31:53Z")
When you view such an export feature, how do you see it working? Lets say you have a cluster of 10s of nodes, with indices with 10s of TB of data. And you want to export an index of several TBs of data, what would the best process be for you? The things to think about are the fact that those indices reside on several different machines, so it needs to be taken into account... On Tue, Aug 23, 2011 at 12:55 AM, Tom Le [dottom@gmail.com](mailto:dottom@gmail.com) wrote: > We are looking at more traditional IT scenarios, such as performing a disk > backup using SAN shadow copy (shut down ES, create snapshot, start ES - we > can copy hundreds of TB's very quickly). > > We also want to move a lot of data from a single standalone node to another > for testing purposes. For example, I have a 10-TB of indexed data that I > maintain as the control data set, and I want multiple researchers to have > their own independent standalone node for testing. Each researcher needs a > different index, where each index is 1-TB in size. With traditional db's, > we'd export and import. For ES, I have to maintain 10 separate instances of > 1-TB each so i can create a large tarball and ship it to a researcher. I'd > like to be able to "export" just specific indexes (making sure mappings and > all other settings are the same). > > There was also a scenario where a client was pointing to the wrong ES > cluster and we indexed 300-GB of data (using a unique index name, so all the > data resided in one index). In this case, I did not have access to the raw > data and would have to write a script to query the cluster, and then > re-index to the correct cluster. An "export" feature here where I could > move an entire index by simply copying files would be nice. > > On Mon, Aug 22, 2011 at 11:43 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > There isn't a way to "import" a specific index, since the fact that its > > "there" in the cluster is also stored in the metadata state of the cluster. > > I agree, it would be nice to have some sort of notion of import / merge, > > but, tricky to get the API right... (in a distributed env, i.e., where is > > the data you plan to import from?) > > > > On Mon, Aug 22, 2011 at 7:57 AM, Tom Le [dottom@gmail.com](mailto:dottom@gmail.com) wrote: > > > > > If a full disk backup was made of an index directory (with ES > > > stopped), is there a way to manually attach or import it? > > > > > > A related question is if there is a way to merge two indexes with the > > > same name, from two separate nodes/clusters? I could write a custom > > > script to read data from cluster 1 and index onto cluster 2, but would > > > be nice if there was a way to simply copy a few TB of data and > > > "import" or "attach" it. ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [August 29, 2011, 7:24pm UTC](https://discuss.elastic.co/t/manually-attaching-an-index/5216/5 "2011-08-29T19:24:44Z")
In our case, we'd like to be able to 1. lock the write to the index, assuming necessary for consistency 2. copy all the shards into a single location (export) along with metadata 3. copy them to new server if the location is not shared 4. import the exported index into the new cluster I realize it may not be as easy as it sounds due to distributed nature but I think this is what's needed. On Monday, August 29, 2011, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > When you view such an export feature, how do you see it working? Lets say > you have a cluster of 10s of nodes, with indices with 10s of TB of data. And > you want to export an index of several TBs of data, what would the best > process be for you? > The things to think about are the fact that those indices reside on > several different machines, so it needs to be taken into account... > > On Tue, Aug 23, 2011 at 12:55 AM, Tom Le [dottom@gmail.com](mailto:dottom@gmail.com) wrote: > > > We are looking at more traditional IT scenarios, such as performing a > > disk backup using SAN shadow copy (shut down ES, create snapshot, start ES - > > we can copy hundreds of TB's very quickly). > > We also want to move a lot of data from a single standalone node to > > another for testing purposes. For example, I have a 10-TB of indexed data > > that I maintain as the control data set, and I want multiple researchers to > > have their own independent standalone node for testing. Each researcher > > needs a different index, where each index is 1-TB in size. With traditional > > db's, we'd export and import. For ES, I have to maintain 10 separate > > instances of 1-TB each so i can create a large tarball and ship it to a > > researcher. I'd like to be able to "export" just specific indexes (making > > sure mappings and all other settings are the same). > > There was also a scenario where a client was pointing to the wrong ES > > cluster and we indexed 300-GB of data (using a unique index name, so all the > > data resided in one index). In this case, I did not have access to the raw > > data and would have to write a script to query the cluster, and then > > re-index to the correct cluster. An "export" feature here where I could > > move an entire index by simply copying files would be nice. > > > > On Mon, Aug 22, 2011 at 11:43 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > There isn't a way to "import" a specific index, since the fact that its > > > "there" in the cluster is also stored in the metadata state of the cluster. > > > I agree, it would be nice to have some sort of notion of import / merge, > > > but, tricky to get the API right... (in a distributed env, i.e., where is > > > the data you plan to import from?) > > > > > > On Mon, Aug 22, 2011 at 7:57 AM, Tom Le [dottom@gmail.com](mailto:dottom@gmail.com) wrote: > > > > > > > If a full disk backup was made of an index directory (with ES > > > > stopped), is there a way to manually attach or import it? > > > > > > > > A related question is if there is a way to merge two indexes with the > > > > same name, from two separate nodes/clusters? I could write a custom > > > > script to read data from cluster 1 and index onto cluster 2, but would > > > > be nice if there was a way to simply copy a few TB of data and > > > > "import" or "attach" it. -- Regards, Berkay Mollamustafaoglu Ph: +1 (571) 766-6292 mberkay on yahoo, google and skype ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:55am UTC](https://discuss.elastic.co/t/manually-attaching-an-index/5216/6 "2017-07-06T03:55:40Z")