# Offline indexing with multiple shards **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [February 9, 2016, 12:24pm UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271 "2016-02-09T12:24:36Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Johannes\_Zillmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johannes_zillmann/32/6939_2.png) [@Johannes\_Zillmann](https://discuss.elastic.co/u/Johannes_Zillmann)\ **Post date:** [February 9, 2016, 12:24pm UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/1 "2016-02-09T12:24:36Z")
I have a question. Is the following possible ? - create multiple indices with multiple ES single node instances - for each index create a snapshot - put all snapshots in one folder and restore them in a ES cluster as a single index Or are there some architectural things that would bite such a procedure, like the document routing !? best Johannes ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 10, 2016, 1:55am UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/2 "2016-02-10T01:55:18Z")
You can do this, the metadata is stored with the snapshot so you can restore across different clusters. Also, did you mean to post this in the hadoop section? ---
**Author:** ![Johannes\_Zillmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johannes_zillmann/32/6939_2.png) [@Johannes\_Zillmann](https://discuss.elastic.co/u/Johannes_Zillmann)\ **Post date:** [February 10, 2016, 10:41am UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/3 "2016-02-10T10:41:02Z")
Ok, good to hear. I think its requiring a little bit of metadata hacking because first we have x indices, each one having exactly one shard and then we need to convert that to 1 index having x shards and the shard count is stored in the metadata.. but it seems doable. I guess i posted it in Hadoop because offline indexing can make a lot of sense in such an environment. Johannes ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 10, 2016, 9:39pm UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/4 "2016-02-10T21:39:22Z")
> [@Johannes\_Zillmann](#): > > I think its requiring a little bit of metadata hacking because first we have x indices, each one having exactly one shard and then we need to convert that to 1 index having x shards and the shard count is stored in the metadata.. but it seems doable. No, that is not possible. It's entirely different to having snapshots from N clusters and then restoring it to a single one. ---
**Author:** ![Johannes\_Zillmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johannes_zillmann/32/6939_2.png) [@Johannes\_Zillmann](https://discuss.elastic.co/u/Johannes_Zillmann)\ **Post date:** [February 10, 2016, 10:54pm UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/5 "2016-02-10T22:54:55Z")
In what problems we would run ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 11, 2016, 2:46am UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/6 "2016-02-11T02:46:58Z")
The problem of it being unachievable. Take a read of [https://www.elastic.co/guide/en/elasticsearch/guide/master/routing-value.html](https://www.elastic.co/guide/en/elasticsearch/guide/master/routing-value.html) ---
**Author:** ![Johannes\_Zillmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johannes_zillmann/32/6939_2.png) [@Johannes\_Zillmann](https://discuss.elastic.co/u/Johannes_Zillmann)\ **Post date:** [February 11, 2016, 8:57am UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/7 "2016-02-11T08:57:36Z")
Ok, makes sense. But if we would apply custom routing it could work right ? We: - also have a fixed number of shards - we can number then from 1..n when creating it - we write our document id with the shard number in it - our custom router routes to the correct shard based on the document id WDYT ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 11, 2016, 8:53pm UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/8 "2016-02-11T20:53:51Z")
Nope. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 11, 2016, 10:01pm UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/9 "2016-02-11T22:01:44Z")
I assume you want to use standalone ES instances in order to reduce the network traffic and ensure that indexing is done locally, is this correct? If so, what you can do, at least as long as you do not need to update your records at a later stage, is to set up all your indexing nodes in a cluster and configure the number of shards so that they have one primary shard each and no replicas. Then index a number of records so that at least one record has been written to each shard. Retrieve the id of one record from each shard and use this as a routing key for all records you want to index into that particular shard. This will give you a single index with a number of shards, while it still allows all documents to be indexed locally only (as long as you provide the correct routing keys). It is not elegant, but I think it should work. ---
**Author:** ![Johannes\_Zillmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johannes_zillmann/32/6939_2.png) [@Johannes\_Zillmann](https://discuss.elastic.co/u/Johannes_Zillmann)\ **Post date:** [February 11, 2016, 11:15pm UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/10 "2016-02-11T23:15:08Z")
Ok, i see. That would eliminate the need for a custom routing function i guess!? There are multiple reasons why we evaluate the offline indexing capabilities of ES. The main thing is speed i guess. Also telling a few other aspects: - we "create" pieces of data in parallel, lets say a map-reduce job - once the job is done and the files are available (in HDFS) we make them available for other parts of our system - similar we want to have the data (duplicated) in ES as well - so producing an index would be a kind of side-effect for the jobs (shouldn't slow down them too much) - since the ES cluster on top of YARN has no reliable persistence, we want to store the indices in hdfs for recovery - we do not need to search the index until its creation has finished nor do we need to update an existing index ---
**Author:** ![Johannes\_Zillmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johannes_zillmann/32/6939_2.png) [@Johannes\_Zillmann](https://discuss.elastic.co/u/Johannes_Zillmann)\ **Post date:** [February 11, 2016, 11:18pm UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/11 "2016-02-11T23:18:55Z")
@warkolm Can you elaborate ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 12, 2016, 12:30am UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/12 "2016-02-12T00:30:50Z")
You cannot just "create" an index with shards outside of Elasticsearch, there are a number of things that it does during this process that doesn't lend itself to this process. You're also not going to do any less work than actually doing it directly in Elasticsearch. There's been other threads where people have asked this in this area as well, it'd be worth checking them. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 12, 2016, 6:18am UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/13 "2016-02-12T06:18:10Z")
Any specific reason you are considering this type of complicated workaround instead of using the existing [ES-Hadoop connector](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/index.html)? ---
**Author:** ![Johannes\_Zillmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johannes_zillmann/32/6939_2.png) [@Johannes\_Zillmann](https://discuss.elastic.co/u/Johannes_Zillmann)\ **Post date:** [February 12, 2016, 11:21am UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/14 "2016-02-12T11:21:39Z")
Thanks @warkolm! @Christian_Dahlqvist our job-pipeline already stands as it is. An ES index would just be an additional product of it! ---
**Author:** ![Johannes\_Zillmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johannes_zillmann/32/6939_2.png) [@Johannes\_Zillmann](https://discuss.elastic.co/u/Johannes_Zillmann)\ **Post date:** [February 15, 2016, 10:44am UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/15 "2016-02-15T10:44:30Z")
Sorry to come back to this... I took the time and evaluated other threads: - [What's the difference between solr and elasticsearch in hdfs case?](https://discuss.elastic.co/t/whats-the-difference-between-solr-and-elasticsearch-in-hdfs-case/19215/7) - [Offline indexing and expected scaling performance](https://discuss.elastic.co/t/offline-indexing-and-expected-scaling-performance/9349) - [How to generate ES index in the hadoop](https://discuss.elastic.co/t/how-to-generate-es-index-in-the-hadoop/12513/7) All have the general advise "Don't do it" but none has an explicit "You can't do it because of x". The only thing mentioned is the routing of documents via document id. But since you providing custom routing values and custom routing functions i see no problem with it. So we are now in an unfortunate situation that our devs saying "Yeah, we have a working prototype for ES offline indexing" and the ES experts saying "Nope, its not possible". So is there something we miss ? Any insights appreciated! Johannes ---
**Author:** ![Johannes\_Zillmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johannes_zillmann/32/6939_2.png) [@Johannes\_Zillmann](https://discuss.elastic.co/u/Johannes_Zillmann)\ **Post date:** [February 18, 2016, 11:11am UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/16 "2016-02-18T11:11:50Z")
Here again a more exact version of what we are -doing. - lets say we have 50 concurrent tasks - each task is spawning its own, unconnected, ES in-vm node - each task creates and index in that nodes, setting the shard count to 50 and through custom routing ensuring that only the one of the 50 shards is filled, which belongs to the task - each task creates an snapshot once the indexing is done - after all tasks has been finished: - a single HDFS folder is created with the snapshot/index metadata (taken from one task) - the one shard which is not empty from each task is copied into that folder - we invoke snapshot restore on our live ES cluster and we have our single index with 50 independently created shards online - when querying the data we always have to specify our custom routing function So this all seem to work pretty well. I'm just wondering if the document id's - which are created outside of the ES cluster concurrently - might potentially collide and cause any Problems... ? @warkolm @costin any thoughts ? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:26pm UTC](https://discuss.elastic.co/t/offline-indexing-with-multiple-shards/41271/17 "2017-07-06T13:26:14Z")