# Design a system that maintains historical view of user activity **URL:** \ **Category:** Elasticsearch\ **Created:** [January 13, 2015, 12:51am UTC](https://discuss.elastic.co/t/design-a-system-that-maintains-historical-view-of-user-activity/21594 "2015-01-13T00:51:55Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Chen\_Wang](https://avatars.discourse-cdn.com/v4/letter/c/b5a626/32.png) [@Chen\_Wang](https://discuss.elastic.co/u/Chen_Wang)\ **Post date:** [January 13, 2015, 12:51am UTC](https://discuss.elastic.co/t/design-a-system-that-maintains-historical-view-of-user-activity/21594/1 "2015-01-13T00:51:55Z")
Hey Guys, I am seeking advice on design a system that maintains a historical view of a user's activities in past one year. Each user can have different activities: email\_open, email\_click, item\_view, add\_to\_cart, purchase etc. The query I would like to do is, for example, Find all customers who browse item A in the past 6 month, and also clicked an email. and I would like the query to be done in reasonable time frame. (for example, within 30 minutes to retrieve 10million such users) Is ES a good candidate for such problem? I am thinking creating an index for each user, but that would have too many indexes(millions). I also tried to index each activity(userid, activity\_type, item\_id, timestamp etc) as a individual document to ES, but it involves join operations which turns out not so efficient(I am using parent-child). Has any of you have experience in designing similar system? As I think this is a rather common problem that need to be solved..(Of cause we can do it in map reduce) Any suggestion is appreciated. Thanks in advance. Chen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ebcb521f-4115-489c-a759-e1459c250eda%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ebcb521f-4115-489c-a759-e1459c250eda%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Mark\_Harwood\_2](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Mark\_Harwood\_2](https://discuss.elastic.co/u/Mark_Harwood_2)\ **Post date:** [January 13, 2015, 8:27am UTC](https://discuss.elastic.co/t/design-a-system-that-maintains-historical-view-of-user-activity/21594/2 "2015-01-13T08:27:01Z")
It was the subject of this talk: [http://www.elasticsearch.org/videos/entity-centric-indexing-london-meetup-sep-2014/](http://www.elasticsearch.org/videos/entity-centric-indexing-london-meetup-sep-2014/) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dc056176-f000-4dd7-b763-95846a240b0d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dc056176-f000-4dd7-b763-95846a240b0d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ed\_Kim](https://avatars.discourse-cdn.com/v4/letter/e/53a042/32.png) [@Ed\_Kim](https://discuss.elastic.co/u/Ed_Kim)\ **Post date:** [January 13, 2015, 8:32pm UTC](https://discuss.elastic.co/t/design-a-system-that-maintains-historical-view-of-user-activity/21594/3 "2015-01-13T20:32:16Z")
The short answer is yes. I've leveraged ES to store events and analyze them in time-based chunks. It's actually a very powerful tool for this type of application. However, you will have to decide on how to model your data to get the most out of it. The first question I would ask is why do you require so many joins? What is the purpose for each join operation, and do you necessarily need to join to everything up front? Can you denormalize some of the data to get what you need in the first pass, and then drill down afterwards? Take into account use experience, and how your queries/model will support that experience. On Monday, January 12, 2015 at 4:51:55 PM UTC-8, Chen Wang wrote: > Hey Guys, > I am seeking advice on design a system that maintains a historical view of > a user's activities in past one year. Each user can have different > activities: email\_open, email\_click, item\_view, add\_to\_cart, purchase etc. > The query I would like to do is, for example, > > Find all customers who browse item A in the past 6 month, and also clicked > an email. > and I would like the query to be done in reasonable time frame. (for > example, within 30 minutes to retrieve 10million such users) > > Is ES a good candidate for such problem? I am thinking creating an index > for each user, but that would have too many indexes(millions). I also tried > to index each activity(userid, activity\_type, item\_id, timestamp etc) as a > individual document to ES, but it involves join operations which turns out > not so efficient(I am using parent-child). > > Has any of you have experience in designing similar system? As I think > this is a rather common problem that need to be solved..(Of cause we can do > it in map reduce) > Any suggestion is appreciated. > > Thanks in advance. > Chen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/22442f86-bb46-4935-bdcb-f1f577665993%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/22442f86-bb46-4935-bdcb-f1f577665993%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:39am UTC](https://discuss.elastic.co/t/design-a-system-that-maintains-historical-view-of-user-activity/21594/4 "2017-07-06T00:39:16Z")