# Bulk Insertion and Aliasing **URL:** \ **Category:** Elasticsearch\ **Created:** [February 5, 2013, 12:30pm UTC](https://discuss.elastic.co/t/bulk-insertion-and-aliasing/10624 "2013-02-05T12:30:02Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![james\_lewis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_lewis/32/2296_2.png) [@james\_lewis](https://discuss.elastic.co/u/james_lewis)\ **Post date:** [February 5, 2013, 12:30pm UTC](https://discuss.elastic.co/t/bulk-insertion-and-aliasing/10624/1 "2013-02-05T12:30:02Z")
Hi guys, I need to do a bulk insert into a cluster but I want to use aliases. Is there any way? I'll be using aliases to route all users data to the same shared when inserting - I think that's the most important thing to be able to do when I do a bulk insert. I noticed there's a routing option on the bulk insert, would this be the way to go? Would you just perform your bulk insert creating a route out of say the user-id and then elsewhere in your application where you do a single insert / search use an alias that routed to the user-id and filtered on the user-id? Cheers, James -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 5, 2013, 1:02pm UTC](https://discuss.elastic.co/t/bulk-insertion-and-aliasing/10624/2 "2013-02-05T13:02:19Z")
Hi James > I need to do a bulk insert into a cluster but I want to use aliases. > Is there any way? Yes, you can use an alias instead of the index name. But for indexing purposes, your alias should only point to one index. For searching, you can use a different alias which points to multiple indices. > I'll be using aliases to route all users data to the same shared when > inserting - I think that's the most important thing to be able to do > when I do a bulk insert. I noticed there's a routing option on the > bulk insert, would this be the way to go? Would you just perform your > bulk insert creating a route out of say the user-id and then elsewhere > in your application where you do a single insert / search use an alias > that routed to the user-id and filtered on the user-id? You can do that, or you can specify a routing and a filter when you create an alias, then you don't need to worry about having to specify it each time. Have a look at > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. and > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![james\_lewis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_lewis/32/2296_2.png) [@james\_lewis](https://discuss.elastic.co/u/james_lewis)\ **Post date:** [February 5, 2013, 1:05pm UTC](https://discuss.elastic.co/t/bulk-insertion-and-aliasing/10624/3 "2013-02-05T13:05:56Z")
Hey cheers for the reply, I think what I'm having trouble with at the moment is how to use the correct routing when performing a bulk insert - so the batch will have 100,000 records in it all from different users. Ultimately, I want to use routing so that 1 users documents are all on the same shard ( a la the user data flow discussed in that video you referenced). Any ideas if that's possible? Regards, James On Tue, Feb 5, 2013 at 1:02 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote: > Hi James > > > I need to do a bulk insert into a cluster but I want to use aliases. > > Is there any way? > > Yes, you can use an alias instead of the index name. But for indexing > purposes, your alias should only point to one index. For searching, you > can use a different alias which points to multiple indices. > > > I'll be using aliases to route all users data to the same shared when > > inserting - I think that's the most important thing to be able to do > > when I do a bulk insert. I noticed there's a routing option on the > > bulk insert, would this be the way to go? Would you just perform your > > bulk insert creating a route out of say the user-id and then elsewhere > > in your application where you do a single insert / search use an alias > > that routed to the user-id and filtered on the user-id? > > You can do that, or you can specify a routing and a filter when you > create an alias, then you don't need to worry about having to specify it > each time. > > Have a look at > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html) > and > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html) > > clint > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 5, 2013, 1:15pm UTC](https://discuss.elastic.co/t/bulk-insertion-and-aliasing/10624/4 "2013-02-05T13:15:29Z")
On Tue, 2013-02-05 at 13:05 +0000, James Lewis wrote: > Hey cheers for the reply, I think what I'm having trouble with at the > moment is how to use the correct routing when performing a bulk insert > > - so the batch will have 100,000 records in it all from different > users. Ultimately, I want to use routing so that 1 users documents > are all on the same shard ( a la the user data flow discussed in that > video you referenced). > Any ideas if that's possible? Yes, that's possible. Either by specifying the routing manually or by using aliases with routing built in (as explained in that video). clint > Regards, > James > > On Tue, Feb 5, 2013 at 1:02 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) > wrote: > Hi James > > ``` > > I need to do a bulk insert into a cluster but I want to use > aliases. > > Is there any way? > > > Yes, you can use an alias instead of the index name. But for > indexing > purposes, your alias should only point to one index. For > searching, you > can use a different alias which points to multiple indices. > > > I'll be using aliases to route all users data to the same > shared when > > inserting - I think that's the most important thing to be > able to do > > when I do a bulk insert. I noticed there's a routing option > on the > > bulk insert, would this be the way to go? Would you just > perform your > > bulk insert creating a route out of say the user-id and then > elsewhere > > in your application where you do a single insert / search > use an alias > > that routed to the user-id and filtered on the user-id? > > > You can do that, or you can specify a routing and a filter > when you > create an alias, then you don't need to worry about having to > specify it > each time. > > Have a look at > http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html > and > http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html > > clint > > -- > You received this message because you are subscribed to the > Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from > it, send an email to elasticsearch > +unsubscribe@googlegroups.com. > For more options, visit > https://groups.google.com/groups/opt_out. > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![james\_lewis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_lewis/32/2296_2.png) [@james\_lewis](https://discuss.elastic.co/u/james_lewis)\ **Post date:** [February 5, 2013, 2:21pm UTC](https://discuss.elastic.co/t/bulk-insertion-and-aliasing/10624/5 "2013-02-05T14:21:08Z")
OK - lets say I create 100000 json documents from some data stored in an SQL database. The documents are in no way grouped so this could either be an entire batch of a single users data, 1 document for 100000 different users or any combination in between. There's no way of knowing. When I insert the 100000 documents I want to make sure that each users data is routed correctly - so a users data will go to 1 shard using a routing based on their id. Every batch of 100000 documents will be as different as the last. I understand that if I insert each document separately I can determine what the user id is and I can use an alias which contains a route based on the id. However, I want to use bulk insert. So what I was looking for was how I might perform the routing (or aliasing) without needing to group my bulk insertions together by user-id or something. In the ES API docs it says you can add the \_routing field per bulk item. So I guess the answer is to do a bit of processing on the client to make sure that for each bulk item, the \_routing field is set correctly. I was having trouble understanding how to do this because it's currently unsupported by my .Net client. Also, the bulk API says nothing about aliases - I guess these should be created after a bulk backfill or on the fly as new documents are added. Cheers! James On Tue, Feb 5, 2013 at 1:15 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote: > On Tue, 2013-02-05 at 13:05 +0000, James Lewis wrote: > > > Hey cheers for the reply, I think what I'm having trouble with at the > > moment is how to use the correct routing when performing a bulk insert > > > > - so the batch will have 100,000 records in it all from different > > users. Ultimately, I want to use routing so that 1 users documents > > are all on the same shard ( a la the user data flow discussed in that > > video you referenced). > > > Any ideas if that's possible? > > Yes, that's possible. Either by specifying the routing manually or by > using aliases with routing built in (as explained in that video). > > clint > > > Regards, > > James > > > > On Tue, Feb 5, 2013 at 1:02 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) > > wrote: > > Hi James > > > > ``` > > > I need to do a bulk insert into a cluster but I want to use > > aliases. > > > Is there any way? > > > > Yes, you can use an alias instead of the index name. But for > > indexing > > purposes, your alias should only point to one index. For > > searching, you > > can use a different alias which points to multiple indices. > > > > > I'll be using aliases to route all users data to the same > > shared when > > > inserting - I think that's the most important thing to be > > able to do > > > when I do a bulk insert. I noticed there's a routing option > > on the > > > bulk insert, would this be the way to go? Would you just > > perform your > > > bulk insert creating a route out of say the user-id and then > > elsewhere > > > in your application where you do a single insert / search > > use an alias > > > that routed to the user-id and filtered on the user-id? > > > > You can do that, or you can specify a routing and a filter > > when you > > create an alias, then you don't need to worry about having to > > specify it > > each time. > > > > Have a look at > > > > ``` > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html) > > > ``` > > and > > > > ``` > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html) > > > ``` > > clint > > > > -- > > You received this message because you are subscribed to the > > Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from > > it, send an email to elasticsearch > > +unsubscribe@googlegroups.com. > > For more options, visit > > https://groups.google.com/groups/opt_out. > > > > ``` > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 5, 2013, 2:34pm UTC](https://discuss.elastic.co/t/bulk-insertion-and-aliasing/10624/6 "2013-02-05T14:34:17Z")
On Tue, 2013-02-05 at 14:21 +0000, James Lewis wrote: > OK - lets say I create 100000 json documents from some data stored in > an SQL database. The documents are in no way grouped so this could > either be an entire batch of a single users data, 1 document for > 100000 different users or any combination in between. There's no way > of knowing. When I insert the 100000 documents I want to make sure > that each users data is routed correctly - so a users data will go to > 1 shard using a routing based on their id. Every batch of 100000 > documents will be as different as the last. curl -XPOST '[http://127.0.0.1:9200/my\_index/couchbaseDocument/\_bulk?pretty=1](http://127.0.0.1:9200/my_index/couchbaseDocument/_bulk?pretty=1)' -d ' {"index" : {"\_id" : 1, "\_routing" : "user\_1", "\_type" : "user"}} {"foo" : "bar"} {"index" : {"\_routing" : "user\_1", "\_type" : "comment"}} {"user\_id" : 1, "foo" : "bar"} ' clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:52am UTC](https://discuss.elastic.co/t/bulk-insertion-and-aliasing/10624/7 "2017-07-06T02:52:57Z")