# How to create user indexes on the fly **URL:** \ **Category:** Elasticsearch\ **Created:** [August 11, 2010, 3:12pm UTC](https://discuss.elastic.co/t/how-to-create-user-indexes-on-the-fly/3206 "2010-08-11T15:12:30Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [August 11, 2010, 3:12pm UTC](https://discuss.elastic.co/t/how-to-create-user-indexes-on-the-fly/3206/1 "2010-08-11T15:12:30Z")
Hi, I have finally got a use-case for per-user indexing. I was wondering what peoples' opinions are on the best way to check and then create an index and type mapping on-the-fly. i.e. 1. get data 2. check if index exists and create if not 3. check if mapping exists and create if not 4. index data How should I go about 2 and 3 in an optimal way. Thanks in advance, Paul (PS I never know when to use indexes or indices) ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://justgiving.com/thetrafalgarway](http://justgiving.com/thetrafalgarway) - 300 miles, 2 bicycles, 36 hours ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [August 11, 2010, 3:29pm UTC](https://discuss.elastic.co/t/how-to-create-user-indexes-on-the-fly/3206/2 "2010-08-11T15:29:45Z")
Hiya > I have finally got a use-case for per-user indexing. I was wondering > what peoples' opinions are on the best way to check and then create an > index and type mapping on-the-fly. Don't forget that each shard is a Lucene instance, so if you have a million users, you will need a LOT of boxes and memory to cope with that. > i.e. > 1. get data > 2. check if index exists and create if not > 3. check if mapping exists and create if not > 4. index data > How should I go about 2 and 3 in an optimal way. You can get a list of all known indices, but that may not be terribly efficient. It may be better to just try it and catch the error, eg: - create the index -\> catch the error if it already exists - put the mapping -\> should just work, if the mapping doesn't exist or hasn't changed clint ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [August 11, 2010, 3:46pm UTC](https://discuss.elastic.co/t/how-to-create-user-indexes-on-the-fly/3206/3 "2010-08-11T15:46:52Z")
Thanks Clint. As always, answers tend to create more questions ! > I have finally got a use-case for per-user indexing. I was wondering > > > what peoples' opinions are on the best way to check and then create an > > index and type mapping on-the-fly. > > Don't forget that each shard is a Lucene instance, so if you have a > million users, you will need a LOT of boxes and memory to cope with > that. Hmm... so are per user indexes not a good idea? We \*are \*expecting lots of users. Will ES definitely keep an instance running for each index even if that index has not been written to or read from for a while? clint > ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://justgiving.com/thetrafalgarway](http://justgiving.com/thetrafalgarway) - 300 miles, 2 bicycles, 36 hours ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [August 11, 2010, 3:54pm UTC](https://discuss.elastic.co/t/how-to-create-user-indexes-on-the-fly/3206/4 "2010-08-11T15:54:46Z")
Hi Paul > ``` > Don't forget that each shard is a Lucene instance, so if you > have a > million users, you will need a LOT of boxes and memory to cope > with > that. > > ``` > > Hmm... so are per user indexes not a good idea? We are expecting lots > of users. Will ES definitely keep an instance running for each index > even if that index has not been written to or read from for a while? "not a good idea" depends on your application, really, but you say that you will have lots of users, so: My understanding is that yes, at least one primary shard must be alive for each index. I say "at least one" although by default you would have more than one. (you can specify this at index creation time) Why not just store a user\_id in each document that needs to be filtered by user? It will be way more efficient. clint ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [August 11, 2010, 4:05pm UTC](https://discuss.elastic.co/t/how-to-create-user-indexes-on-the-fly/3206/5 "2010-08-11T16:05:24Z")
Another thought (and apologies for confusing the conversation by adding another Paul), if you need per user mapping is to have a single index w/ as many shards as needed, and each user can be their own document type, as mappings are defined per document type. Regards, Paul P On Aug 11, 9:54 am, Clinton Gormley [clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) wrote: > Hi Paul > > > ``` > > Don't forget that each shard is a Lucene instance, so if you > > have a > > million users, you will need a LOT of boxes and memory to cope > > with > > that. > > > > ``` > > > Hmm... so are per user indexes not a good idea? We are expecting lots > > of users. Will ES definitely keep an instance running for each index > > even if that index has not been written to or read from for a while? > > "not a good idea" depends on your application, really, but you say that > you will have lots of users, so: > > My understanding is that yes, at least one primary shard must be alive > for each index. I say "at least one" although by default you would have > more than one. (you can specify this at index creation time) > > Why not just store a user\_id in each document that needs to be filtered > by user? It will be way more efficient. > > clint ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [August 11, 2010, 4:15pm UTC](https://discuss.elastic.co/t/how-to-create-user-indexes-on-the-fly/3206/6 "2010-08-11T16:15:36Z")
Ah, that's interesting. Any overheads doing this rather than the adding a userId suggestion? I guess this will make searches quick as by using the type you're essencially pre-filtering. Thanks Paul and Clint! On Wed, Aug 11, 2010 at 5:05 PM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > Another thought (and apologies for confusing the conversation by > adding another Paul), if you need per user mapping is to have a single > index w/ as many shards as needed, and each user can be their own > document type, as mappings are defined per document type. > > Regards, > Paul P > > On Aug 11, 9:54 am, Clinton Gormley [clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) wrote: > > > Hi Paul > > > > > ``` > > > Don't forget that each shard is a Lucene instance, so if you > > > have a > > > million users, you will need a LOT of boxes and memory to cope > > > with > > > that. > > > > > > ``` > > > > > Hmm... so are per user indexes not a good idea? We are expecting lots > > > of users. Will ES definitely keep an instance running for each index > > > even if that index has not been written to or read from for a while? > > > > "not a good idea" depends on your application, really, but you say that > > you will have lots of users, so: > > > > My understanding is that yes, at least one primary shard must be alive > > for each index. I say "at least one" although by default you would have > > more than one. (you can specify this at index creation time) > > > > Why not just store a user\_id in each document that needs to be filtered > > by user? It will be way more efficient. > > > > clint ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [Edward Burge is fundraising for Help for Heroes](http://justgiving.com/thetrafalgarway) - 300 miles, 2 bicycles, 36 hours ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 11, 2010, 5:04pm UTC](https://discuss.elastic.co/t/how-to-create-user-indexes-on-the-fly/3206/7 "2010-08-11T17:04:51Z")
Hi, Its basically the same, the multi type support within an index is done by using an \_type field to each indexed document and automatically filtering by it when applicable. -shay.banon On Wed, Aug 11, 2010 at 7:15 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > Ah, that's interesting. Any overheads doing this rather than the adding a > userId suggestion? > > I guess this will make searches quick as by using the type you're > essencially pre-filtering. > > Thanks Paul and Clint! > > On Wed, Aug 11, 2010 at 5:05 PM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > > > Another thought (and apologies for confusing the conversation by > > adding another Paul), if you need per user mapping is to have a single > > index w/ as many shards as needed, and each user can be their own > > document type, as mappings are defined per document type. > > > > Regards, > > Paul P > > > > On Aug 11, 9:54 am, Clinton Gormley [clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) wrote: > > > > > Hi Paul > > > > > > > ``` > > > > Don't forget that each shard is a Lucene instance, so if you > > > > have a > > > > million users, you will need a LOT of boxes and memory to cope > > > > with > > > > that. > > > > > > > > ``` > > > > > > > Hmm... so are per user indexes not a good idea? We are expecting lots > > > > of users. Will ES definitely keep an instance running for each index > > > > even if that index has not been written to or read from for a while? > > > > > > "not a good idea" depends on your application, really, but you say that > > > you will have lots of users, so: > > > > > > My understanding is that yes, at least one primary shard must be alive > > > for each index. I say "at least one" although by default you would have > > > more than one. (you can specify this at index creation time) > > > > > > Why not just store a user\_id in each document that needs to be filtered > > > by user? It will be way more efficient. > > > > > > clint > > ## -- > > Paul Loy > [paul@keteracel.com](mailto:paul@keteracel.com) > [Edward Burge is fundraising for Help for Heroes](http://justgiving.com/thetrafalgarway) - 300 miles, 2 bicycles, 36 hours ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:20am UTC](https://discuss.elastic.co/t/how-to-create-user-indexes-on-the-fly/3206/8 "2017-07-06T04:20:56Z")