# Parent/Child and index allocation problem **URL:** \ **Category:** Elasticsearch\ **Created:** [July 25, 2013, 10:11am UTC](https://discuss.elastic.co/t/parent-child-and-index-allocation-problem/12934 "2013-07-25T10:11:44Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Thomas\_Bolis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_bolis/32/700_2.png) [@Thomas\_Bolis](https://discuss.elastic.co/u/Thomas_Bolis)\ **Post date:** [July 25, 2013, 10:11am UTC](https://discuss.elastic.co/t/parent-child-and-index-allocation-problem/12934/1 "2013-07-25T10:11:44Z")
Hi, I wanted to ask a best practice for the following problem: I have three types of documents: - User - IncomingLog - OutgoingLog For my mapping I have setup the \*User \*to be the parent of the other two documents \*IncomingLog \*and _OutgoingLog_. Users are my user base whereas log documents are historical data. I do this to perform various parent/child queries to fetch users that match specific criteria based on their log history. What I'm wondering is that after some time the logs will become huge therefore I need to do some time flow of data by changing and index every lets say month for log documents. What is the best possible way of doing that? given that the Users are not historical data and i cannot change index along with the log docs. I know that parent/child documents are sharded in the same shard, can we say that we can have one index for the users, without creating a new one each month but have an other index for the logs and creating a new one each month and combining them with an alias?? My users are way to many (I think) around 10 milliion or even more, to create one index per user, and it does not seems obvious to me if this is a good thing to do. What do you suggest?? Thank you in advance. Thomas -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [July 25, 2013, 10:39am UTC](https://discuss.elastic.co/t/parent-child-and-index-allocation-problem/12934/2 "2013-07-25T10:39:00Z")
Hello Thomas, I think you can keep all the data in a single index, and use the TTL field[http://www.elasticsearch.org/guide/reference/mapping/ttl-field/](http://www.elasticsearch.org/guide/reference/mapping/ttl-field/)to get rid of the logs after a while. Sure, from a performance perspective, it would be nice to have time-based indices for logs. Because searching in recent data and deleting will be more efficient. But you have to have users in the same index as the logs to be able to make the parent-child relationship. You can still make that happen, but I don't see a straightforward way. For example, you can have a master index with all your users, and time-based indices with your logs. Whenever you index a new log, you can make sure the user it belongs to is indexed in the same index as well. Or, you can duplicate all your users in each time-based index. Best regards, Radu On Thu, Jul 25, 2013 at 1:11 PM, Thomas [thomas.bolis@gmail.com](mailto:thomas.bolis@gmail.com) wrote: > Hi, > > I wanted to ask a best practice for the following problem: > > I have three types of documents: > > - User > - IncomingLog > - OutgoingLog > > For my mapping I have setup the \*User \*to be the parent of the other two > documents \*IncomingLog \*and _OutgoingLog_. Users are my user base whereas > log documents are historical data. I do this to perform various > parent/child queries to fetch users that match specific criteria based on > their log history. > > What I'm wondering is that after some time the logs will become huge > therefore I need to do some time flow of data by changing and index every > lets say month for log documents. What is the best possible way of doing > that? given that the Users are not historical data and i cannot change > index along with the log docs. I know that parent/child documents are > sharded in the same shard, can we say that we can have one index for the > users, without creating a new one each month but have an other index for > the logs and creating a new one each month and combining them with an > alias?? My users are way to many (I think) around 10 milliion or even more, > to create one index per user, and it does not seems obvious to me if this > is a good thing to do. > > What do you suggest?? > > Thank you in advance. > Thomas > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Thomas\_Bolis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_bolis/32/700_2.png) [@Thomas\_Bolis](https://discuss.elastic.co/u/Thomas_Bolis)\ **Post date:** [July 25, 2013, 2:13pm UTC](https://discuss.elastic.co/t/parent-child-and-index-allocation-problem/12934/3 "2013-07-25T14:13:18Z")
Dear Radu, Thanks for your reply, unfortunately this does not solve my case. What i was thinking is split my users in groups in way where a user will belong strictly to a group so I can have some short of a load balancing of users along with their logs to a set of identical indexes and search across all indexes with an alias. But again this will partially solve my problem as the number of indexes will be fixed. Does this makes sense? T. On Thu, Jul 25, 2013 at 1:39 PM, Radu Gheorghe [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com)wrote: > Hello Thomas, > > I think you can keep all the data in a single index, and use the TTL field[http://www.elasticsearch.org/guide/reference/mapping/ttl-field/](http://www.elasticsearch.org/guide/reference/mapping/ttl-field/)to get rid of the logs after a while. > > Sure, from a performance perspective, it would be nice to have time-based > indices for logs. Because searching in recent data and deleting will be > more efficient. But you have to have users in the same index as the logs to > be able to make the parent-child relationship. You can still make that > happen, but I don't see a straightforward way. > > For example, you can have a master index with all your users, and > time-based indices with your logs. Whenever you index a new log, you can > make sure the user it belongs to is indexed in the same index as well. Or, > you can duplicate all your users in each time-based index. > > Best regards, > Radu > > On Thu, Jul 25, 2013 at 1:11 PM, Thomas [thomas.bolis@gmail.com](mailto:thomas.bolis@gmail.com) wrote: > > > Hi, > > > > I wanted to ask a best practice for the following problem: > > > > I have three types of documents: > > > > - User > > - IncomingLog > > - OutgoingLog > > > > For my mapping I have setup the \*User \*to be the parent of the other two > > documents \*IncomingLog \*and _OutgoingLog_. Users are my user base > > whereas log documents are historical data. I do this to perform various > > parent/child queries to fetch users that match specific criteria based on > > their log history. > > > > What I'm wondering is that after some time the logs will become huge > > therefore I need to do some time flow of data by changing and index every > > lets say month for log documents. What is the best possible way of doing > > that? given that the Users are not historical data and i cannot change > > index along with the log docs. I know that parent/child documents are > > sharded in the same shard, can we say that we can have one index for the > > users, without creating a new one each month but have an other index for > > the logs and creating a new one each month and combining them with an > > alias?? My users are way to many (I think) around 10 milliion or even more, > > to create one index per user, and it does not seems obvious to me if this > > is a good thing to do. > > > > What do you suggest?? > > > > Thank you in advance. > > > > Thomas > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/wjyE9Nu2vjI/unsubscribe](https://groups.google.com/d/topic/elasticsearch/wjyE9Nu2vjI/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:24am UTC](https://discuss.elastic.co/t/parent-child-and-index-allocation-problem/12934/4 "2017-07-06T02:24:33Z")