# \[Ann\] Easy importing/exporting indices with the Knapsack plugin **URL:** \ **Category:** Elasticsearch\ **Created:** [December 9, 2012, 4:07pm UTC](https://discuss.elastic.co/t/ann-easy-importing-exporting-indices-with-the-knapsack-plugin/10003 "2012-12-09T16:07:16Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [December 9, 2012, 4:07pm UTC](https://discuss.elastic.co/t/ann-easy-importing-exporting-indices-with-the-knapsack-plugin/10003/1 "2012-12-09T16:07:16Z")
Hi, I have written a very simple and easy-to-use importing/exporting tool for Elasticsearch, the Knapsack plugin. How does it work? Knapsack depends on the \_source field being enabled. By executing an \_export command via curl HTTP POST, the export starts and performs a background threaded loop of scan/scroll request/response over the index. Each document \_source is written as a tar entry to a tar archive file in the local file system. By default, the tar archive is gzipped. Also, the index setting and mapping is written into the tar. The tar entry name consists of index/type/id, and the name of the tar archive consists of '\_' '.tar.gz' Importing works exactly the other way round. Put a tar archive into the file system, and import it via \_import with curl HTTP POST. A background thread is spawned and the tar archive is bulk-inserted into Elasticsearch. By file renaming, you can even copy indices. Note, it is not a full-fledged backup/restore solution, it is just a toy, with many features not (yet) present. It can help in situations where you want to offload indices quickly and transfer it to another place, e.g. for bug hunting, or for testing. So, if your indices want to take a hike, maybe across Elasticsearch clusters, put the knapsack on and enjoy! > **[jprante/elasticsearch-knapsack](https://github.com/jprante/elasticsearch-knapsack)** > > elasticsearch-knapsack - Knapsack plugin is an import/export tool for Elasticsearch Cheers, Jörg -- ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 9, 2012, 8:53pm UTC](https://discuss.elastic.co/t/ann-easy-importing-exporting-indices-with-the-knapsack-plugin/10003/2 "2012-12-09T20:53:55Z")
Very nice. I will play with it! -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 9 déc. 2012 à 17:07, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) a écrit : Hi, I have written a very simple and easy-to-use importing/exporting tool for Elasticsearch, the Knapsack plugin. How does it work? Knapsack depends on the \_source field being enabled. By executing an \_export command via curl HTTP POST, the export starts and performs a background threaded loop of scan/scroll request/response over the index. Each document \_source is written as a tar entry to a tar archive file in the local file system. By default, the tar archive is gzipped. Also, the index setting and mapping is written into the tar. The tar entry name consists of index/type/id, and the name of the tar archive consists of '\_' '.tar.gz' Importing works exactly the other way round. Put a tar archive into the file system, and import it via \_import with curl HTTP POST. A background thread is spawned and the tar archive is bulk-inserted into Elasticsearch. By file renaming, you can even copy indices. Note, it is not a full-fledged backup/restore solution, it is just a toy, with many features not (yet) present. It can help in situations where you want to offload indices quickly and transfer it to another place, e.g. for bug hunting, or for testing. So, if your indices want to take a hike, maybe across Elasticsearch clusters, put the knapsack on and enjoy! > **[jprante/elasticsearch-knapsack](https://github.com/jprante/elasticsearch-knapsack)** > > elasticsearch-knapsack - Knapsack plugin is an import/export tool for Elasticsearch Cheers, Jörg -- -- ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [December 10, 2012, 4:40am UTC](https://discuss.elastic.co/t/ann-easy-importing-exporting-indices-with-the-knapsack-plugin/10003/3 "2012-12-10T04:40:32Z")
Hi Jörg, Very nice. How come you went for export+import and not a straight scan/scroll=\>index? ## Thanks, Otis ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) On Sunday, December 9, 2012 11:07:16 AM UTC-5, Jörg Prante wrote: > Hi, > > I have written a very simple and easy-to-use importing/exporting tool for > Elasticsearch, the Knapsack plugin. > > How does it work? > > Knapsack depends on the \_source field being enabled. By executing an > \_export command via curl HTTP POST, the export starts and performs a > background threaded loop of scan/scroll request/response over the index. > Each document \_source is written as a tar entry to a tar archive file in > the local file system. By default, the tar archive is gzipped. > > Also, the index setting and mapping is written into the tar. > > The tar entry name consists of index/type/id, and the name of the tar > archive consists of '\_' '.tar.gz' > > Importing works exactly the other way round. Put a tar archive into the > file system, and import it via \_import with curl HTTP POST. A background > thread is spawned and the tar archive is bulk-inserted into Elasticsearch. > By file renaming, you can even copy indices. > > Note, it is not a full-fledged backup/restore solution, it is just a toy, > with many features not (yet) present. It can help in situations where you > want to offload indices quickly and transfer it to another place, e.g. for > bug hunting, or for testing. > > So, if your indices want to take a hike, maybe across Elasticsearch > clusters, put the knapsack on and enjoy! > > [GitHub - jprante/elasticsearch-knapsack: Knapsack plugin is an import/export tool for Elasticsearch](https://github.com/jprante/elasticsearch-knapsack) > > Cheers, > > Jörg -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [December 10, 2012, 5:32pm UTC](https://discuss.elastic.co/t/ann-easy-importing-exporting-indices-with-the-knapsack-plugin/10003/4 "2012-12-10T17:32:27Z")
Well, a straight scan/scroll-\>index is already there, in the reindex plugin 🙂 > **[GitHub - karussell/elasticsearch-reindex: Simple re-indexing. To backup,...](https://github.com/karussell/elasticsearch-reindex)** > > Simple re-indexing. To backup, apply index settings changes and more ElasticMagic - GitHub - karussell/elasticsearch-reindex: Simple re-indexing. To backup, apply index settings changes and more El... My primary intention was being able to create tar archive files from an index, for transporting the archive to another Elasticsearch instance. Jörg On Monday, December 10, 2012 5:40:32 AM UTC+1, Otis Gospodnetic wrote: > Hi Jörg, > > Very nice. How come you went for export+import and not a straight > scan/scroll=\>index? > > ## Thanks, Otis > > ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > > On Sunday, December 9, 2012 11:07:16 AM UTC-5, Jörg Prante wrote: > > > Hi, > > > > I have written a very simple and easy-to-use importing/exporting tool for > > Elasticsearch, the Knapsack plugin. > > > > How does it work? > > > > Knapsack depends on the \_source field being enabled. By executing an > > \_export command via curl HTTP POST, the export starts and performs a > > background threaded loop of scan/scroll request/response over the index. > > Each document \_source is written as a tar entry to a tar archive file in > > the local file system. By default, the tar archive is gzipped. > > > > Also, the index setting and mapping is written into the tar. > > > > The tar entry name consists of index/type/id, and the name of the tar > > archive consists of '\_' '.tar.gz' > > > > Importing works exactly the other way round. Put a tar archive into the > > file system, and import it via \_import with curl HTTP POST. A background > > thread is spawned and the tar archive is bulk-inserted into Elasticsearch. > > By file renaming, you can even copy indices. > > > > Note, it is not a full-fledged backup/restore solution, it is just a toy, > > with many features not (yet) present. It can help in situations where you > > want to offload indices quickly and transfer it to another place, e.g. for > > bug hunting, or for testing. > > > > So, if your indices want to take a hike, maybe across Elasticsearch > > clusters, put the knapsack on and enjoy! > > > > [GitHub - jprante/elasticsearch-knapsack: Knapsack plugin is an import/export tool for Elasticsearch](https://github.com/jprante/elasticsearch-knapsack) > > > > Cheers, > > > > Jörg -- ---
**Author:** ![Jeff\_Rick](https://avatars.discourse-cdn.com/v4/letter/j/e9c0ed/32.png) [@Jeff\_Rick](https://discuss.elastic.co/u/Jeff_Rick)\ **Post date:** [December 11, 2012, 12:09am UTC](https://discuss.elastic.co/t/ann-easy-importing-exporting-indices-with-the-knapsack-plugin/10003/5 "2012-12-11T00:09:42Z")
So I have been looking to do something like this but was thinking I would just copy the local directory structures from one cluster to another. As long as the number of nodes match it should just work. Did you look into this? Thanks -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [December 11, 2012, 1:01am UTC](https://discuss.elastic.co/t/ann-easy-importing-exporting-indices-with-the-knapsack-plugin/10003/6 "2012-12-11T01:01:56Z")
Of course I looked into the copy approach. But that approach has some limitations. Beside the same number of nodes, you need the same ES version, and the guarantee that local directory structures had been flushed into a consistent state. And, you have binary Lucene indexes plus ES control files, which is somewhat cryptic to get opened and processed by non-Lucene tools. My challenge is offloading JSON data for postprocessing, and also moving it to other ES installations for reloading, no matter how many nodes or what version 🙂 Imagine the following use cases: create index dumps for remote analysis, preparing a set of index data for reproducible benchmarks, or packaging aggregated data for further processing outside ES which has been harvested from many places by many methods over a while. Jörg -- ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [December 11, 2012, 5:13am UTC](https://discuss.elastic.co/t/ann-easy-importing-exporting-indices-with-the-knapsack-plugin/10003/7 "2012-12-11T05:13:38Z")
Jörg, I think this is great plugin! I can think of interesting use cases. Thanks for sharing. Lukáš Dne 11.12.2012 2:01 "Jörg Prante" [joergprante@gmail.com](mailto:joergprante@gmail.com) napsal(a): > Of course I looked into the copy approach. But that approach has some > limitations. > > Beside the same number of nodes, you need the same ES version, and the > guarantee that local directory structures had been flushed into a > consistent state. And, you have binary Lucene indexes plus ES control > files, which is somewhat cryptic to get opened and processed by non-Lucene > tools. > > My challenge is offloading JSON data for postprocessing, and also moving > it to other ES installations for reloading, no matter how many nodes or > what version 🙂 > > Imagine the following use cases: create index dumps for remote analysis, > preparing a set of index data for reproducible benchmarks, or packaging > aggregated data for further processing outside ES which has been harvested > from many places by many methods over a while. > > Jörg > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:00am UTC](https://discuss.elastic.co/t/ann-easy-importing-exporting-indices-with-the-knapsack-plugin/10003/8 "2017-07-06T03:00:26Z")