# Index design for web activity **URL:** \ **Category:** Elasticsearch\ **Created:** [December 19, 2014, 11:53pm UTC](https://discuss.elastic.co/t/index-design-for-web-activity/21338 "2014-12-19T23:53:48Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Chen\_Wang](https://avatars.discourse-cdn.com/v4/letter/c/b5a626/32.png) [@Chen\_Wang](https://discuss.elastic.co/u/Chen_Wang)\ **Post date:** [December 19, 2014, 11:53pm UTC](https://discuss.elastic.co/t/index-design-for-web-activity/21338/1 "2014-12-19T23:53:48Z")
Hey Guys, Wanna seek your suggestions on the index design for web activities. Lets say I have browse data, online purchase data, and store purchase data, and I will need to save a year of them. For browse data, a year of data is around 80G , online purchase data is around 50G, and offline data is around 1T. I have to do query like, e.g, find all the customers who browsed item A in the past X months, and also online purchased B in the past Y month. Originally I am using complicated parent/child structure, and that sometimes results in very bad performance. and I store all browse data/online purchase/store purchase in one index distributed to 7 shards. I have 7 machines with 128G each, and 1T hard disk. Now, I am trying to save each of those type of data into its own index, say browse\_v1, onlinepurchase\_v1, storepurchase\_v1. Since its time based data, how should I decide to break them into monthly , or simply yearly? for browse(70G)/online purchase(50G), i think i can just use one index and one shard for them,. or should I break them into monthly data instead? breaking into monthly indexes gives me the flexibility of adding/removing data, but it also will decrease the query performance, right? (search against 1 index now becomes search against 12 indexes). For store data(1T) apparently I have to break them into at least monthly index, but each monthly index still contains around 100G data. With my current cluster, how many shards should I allocate to each monthly index? I am also concerned about the query performance. Then since I am now storing them into separate indexes, to achieve the query I want, I will need to do application level join. Is this the common way to handle such user case? I know I should perform some testing first, but hope someone may have similar experience in handling this and could provide some guidance. thanks in advance, Chen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2cba8839-2577-4fd7-b1e9-550ae579bb1a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2cba8839-2577-4fd7-b1e9-550ae579bb1a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [December 21, 2014, 9:54pm UTC](https://discuss.elastic.co/t/index-design-for-web-activity/21338/2 "2014-12-21T21:54:42Z")
On Sat, Dec 20, 2014 at 12:53 AM, Chen Wang [chen.apache.solr@gmail.com](mailto:chen.apache.solr@gmail.com) wrote: > Hey Guys, > Wanna seek your suggestions on the index design for web activities. > Lets say I have browse data, online purchase data, and store purchase > data, and I will need to save a year of them. > For browse data, a year of data is around 80G , online purchase data is > around 50G, and offline data is around 1T. > > I have to do query like, e.g, find all the customers who browsed item A in > the past X months, and also online purchased B in the past Y month. > Originally I am using complicated parent/child structure, and that > sometimes results in very bad performance. and I store all browse > data/online purchase/store purchase in one index distributed to 7 shards. Parent/child is indeed slow. Can you somehow denormalize your data to make queries faster? > I have 7 machines with 128G each, and 1T hard disk. > > Now, I am trying to save each of those type of data into its own index, > say browse\_v1, onlinepurchase\_v1, storepurchase\_v1. Since its time based > data, how should I decide to break them into monthly , or simply yearly? > for browse(70G)/online purchase(50G), i think i can just use one index and > one shard for them,. or should I break them into monthly data instead? > breaking into monthly indexes gives me the flexibility of adding/removing > data, but it also will decrease the query performance, right? (search > against 1 index now becomes search against 12 indexes). > > For store data(1T) apparently I have to break them into at least monthly > index, but each monthly index still contains around 100G data. With my > current cluster, how many shards should I allocate to each monthly index? I > am also concerned about the query performance. > > Then since I am now storing them into separate indexes, to achieve the > query I want, I will need to do application level join. Is this the common > way to handle such user case? As much as possible, you should try to design you documents in such a way that you don't need to perform joins at search time. Would it be possible for you to adopt a more "entity-centric" approach at indexing time? > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. > I know I should perform some testing first, but hope someone may have > similar experience in handling this and could provide some guidance. The Elasticsearch book has a chapter about "designing for scale" that gives good advices around modeling the data and chosing the right shard size and numbers of shards: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5m3NXxXhj1EZCJ-H\_3fc0kyjiJCqXKNBqfYbcXo3Mxdg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5m3NXxXhj1EZCJ-H_3fc0kyjiJCqXKNBqfYbcXo3Mxdg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Chen\_Wang](https://avatars.discourse-cdn.com/v4/letter/c/b5a626/32.png) [@Chen\_Wang](https://discuss.elastic.co/u/Chen_Wang)\ **Post date:** [January 14, 2015, 6:06pm UTC](https://discuss.elastic.co/t/index-design-for-web-activity/21338/3 "2015-01-14T18:06:18Z")
Adrien, Is there a more clear version of the video record? I can barely see the slides, and don't quite get the idea of entity-centric.. Does it mean, for my user case, to maintain a single user document that contains list of activities, and during the index time, just simply update the list of this property? something like: { \_source:{ customer\_id: 123 browse: [{item1, time1},{item2, time2}], purchase: [{item1,time1},{item2, time2}], } } during the index time, I just update the browse/purchase list? Then my query basically becomes flat. Is my understanding correct? Chen On Sunday, December 21, 2014 at 1:54:48 PM UTC-8, Adrien Grand wrote: > On Sat, Dec 20, 2014 at 12:53 AM, Chen Wang \<[chen.apa...@gmail.com](mailto:chen.apa...@gmail.com) > \\> wrote: > > > Hey Guys, > > Wanna seek your suggestions on the index design for web activities. > > Lets say I have browse data, online purchase data, and store purchase > > data, and I will need to save a year of them. > > For browse data, a year of data is around 80G , online purchase data is > > around 50G, and offline data is around 1T. > > > > I have to do query like, e.g, find all the customers who browsed item A > > in the past X months, and also online purchased B in the past Y month. > > Originally I am using complicated parent/child structure, and that > > sometimes results in very bad performance. and I store all browse > > data/online purchase/store purchase in one index distributed to 7 shards. > > Parent/child is indeed slow. Can you somehow denormalize your data to make > queries faster? > > > I have 7 machines with 128G each, and 1T hard disk. > > > > Now, I am trying to save each of those type of data into its own index, > > say browse\_v1, onlinepurchase\_v1, storepurchase\_v1. Since its time based > > data, how should I decide to break them into monthly , or simply yearly? > > for browse(70G)/online purchase(50G), i think i can just use one index and > > one shard for them,. or should I break them into monthly data instead? > > breaking into monthly indexes gives me the flexibility of adding/removing > > data, but it also will decrease the query performance, right? (search > > against 1 index now becomes search against 12 indexes). > > > > For store data(1T) apparently I have to break them into at least monthly > > index, but each monthly index still contains around 100G data. With my > > current cluster, how many shards should I allocate to each monthly index? I > > am also concerned about the query performance. > > > > Then since I am now storing them into separate indexes, to achieve the > > query I want, I will need to do application level join. Is this the common > > way to handle such user case? > > As much as possible, you should try to design you documents in such a way > that you don't need to perform joins at search time. Would it be possible > for you to adopt a more "entity-centric" approach at indexing time? > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/entity-centric-indexing-london-meetup-sep-2014/) > > > I know I should perform some testing first, but hope someone may have > > similar experience in handling this and could provide some guidance. > > The Elasticsearch book has a chapter about "designing for scale" that > gives good advices around modeling the data and chosing the right shard > size and numbers of shards: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/scale.html) > > -- > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9c6ac540-2d77-49de-85b4-7fd1574ff2ed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9c6ac540-2d77-49de-85b4-7fd1574ff2ed%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:38am UTC](https://discuss.elastic.co/t/index-design-for-web-activity/21338/4 "2017-07-06T00:38:55Z")