# Handling date-based-indexing and parent/child relationships **URL:** \ **Category:** Elasticsearch\ **Created:** [July 7, 2017, 12:50pm UTC](https://discuss.elastic.co/t/handling-date-based-indexing-and-parent-child-relationships/92269 "2017-07-07T12:50:01Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Aaron3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron3/32/19703_2.png) [@Aaron3](https://discuss.elastic.co/u/Aaron3)\ **Post date:** [July 7, 2017, 12:50pm UTC](https://discuss.elastic.co/t/handling-date-based-indexing-and-parent-child-relationships/92269/1 "2017-07-07T12:50:01Z")
We are using Elastic for realtime web analytics. Because we don't care about long-term data in ES, we need a way to delete old data easily, the normal solution to this to use a data-based index, so old indexes are deleted every X weeks. However, this means that every time the index increments, we lose the ability to query all parent/child relationships between documents on one side of the increment vs the other. A parent hit that happens a midnight will be in a different index than its child events. We use parent-child relationships between session/pageview and pageview/event with the following basic relationship: - a Session has many Pageviews which have many Events. - session/pageview are mostly denormalized - pageview/event are completely normalized - we need to support querying events using the parent/child queries We are trying to determine the best solution. - Because of the scale of data, we can not denormalize hits/events. Each hit can have dozens of events. (We have a separate index of raw server logs and it takes up 5-6 times as much disk space) - Week-based or monthly indexes cause the same problem less often - A single massive index is not a great solution _Right now, it seems our best solution is storing the event into the index of the hit based on the hit timestamp. Are there any other good solutions?_ Notes: - Our indexes are version/date based. If we push a version change, a new index is created but backwards incompatible version changes are rare - PageViews are relatively short lived: measured in minutes/hours. - The majority of sessions are measured in hours/days, but some can be weeks/months (users who do not close their browsers.) - Because it's a minority of data and we use denormalized data, we are okay losing some parent sessions. - We process all of our incoming data to files, then injest with logstash to handle basic transforms like changing `session_id` -\> `_parent` etc. - The scale of data is "small", 500m documents a month, though this will probably triple once all the kinks are worked out ---
**Author:** ![Aaron3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron3/32/19703_2.png) [@Aaron3](https://discuss.elastic.co/u/Aaron3)\ **Post date:** [July 7, 2017, 3:39pm UTC](https://discuss.elastic.co/t/handling-date-based-indexing-and-parent-child-relationships/92269/2 "2017-07-07T15:39:42Z")
And as an addendum, with the removal of `_type` in 6.0, how will this work with the new `join` syntax in 5.6/6.0? I see joins are still same-shard, but indexes can only have one `_type`? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 11, 2017, 1:34am UTC](https://discuss.elastic.co/t/handling-date-based-indexing-and-parent-child-relationships/92269/3 "2017-07-11T01:34:52Z")
The choices here are to flatten everything, so each event has all the info required to drop the use of parent/child. Or to have two indices, one for the events and one for the entities, and then join within your app/code. Parent/Child will have a special field to join on, that's all `_type` is anyway. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 8, 2017, 1:35am UTC](https://discuss.elastic.co/t/handling-date-based-indexing-and-parent-child-relationships/92269/4 "2017-08-08T01:35:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.