# E-commerce logging strategy **URL:** \ **Category:** Elasticsearch\ **Created:** [February 12, 2020, 1:55pm UTC](https://discuss.elastic.co/t/e-commerce-logging-strategy/219003 "2020-02-12T13:55:58Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Chaim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chaim/32/62552_2.png) [@Chaim](https://discuss.elastic.co/u/Chaim)\ **Post date:** [February 12, 2020, 1:55pm UTC](https://discuss.elastic.co/t/e-commerce-logging-strategy/219003/1 "2020-02-12T13:55:59Z")
**Our Current Setup** - We have an elastic cloud deployment, collecting browsing logs for an e-commerce website, on a regular day our log reaches ~100GB - Given the huge amount of data, we only keep 3 days of logs then archive it. - We also set up a rollup index as 5 min intervals, grouped by url, status-code, RTT etc. the rollup log results in ~5GB data daily - We would like to keep the rollup logs for at least 2 years - The problem is that after amount its over 100GB and did not find yet a way to automatically rollover... - So far we do a new rollup setup monthly. so we get it in a new log. **My question is** Does our setup make sense? 1. Can you offer some advice on doing things better 2. Is there any reference / example of a good ecomm logging strategy ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [February 14, 2020, 9:03am UTC](https://discuss.elastic.co/t/e-commerce-logging-strategy/219003/2 "2020-02-14T09:03:05Z")
Hey, have you seen the [rollover API](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/indices-rollover-index.html) - you might want to take a look at [index lifecycle management](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/getting-started-index-lifecycle-management.html) and frozen indices as well in order to reduce memory footprint. Hope this helps as a start. --Alex ---
**Author:** ![Chaim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chaim/32/62552_2.png) [@Chaim](https://discuss.elastic.co/u/Chaim)\ **Post date:** [February 14, 2020, 1:51pm UTC](https://discuss.elastic.co/t/e-commerce-logging-strategy/219003/3 "2020-02-14T13:51:33Z")
Thanks @spinscale! sure, we're actually using the rollover API for our regular indexes, thru the "index lifecycle management". the challenge we have is combining the 2... as far as we see, the "rollup\_index": "" (in the rollup definition) needs to be set to a static index name (no variable or alias, we tried either and it failed).. so again to clarify, we love the ILP idea, and want to extand this "set and forget" concept also to our rollup's but we are stuck here... does my question make more sense now? ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [February 17, 2020, 10:31am UTC](https://discuss.elastic.co/t/e-commerce-logging-strategy/219003/4 "2020-02-17T10:31:24Z")
These logs are not enough. Please see [https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html#rpm-running-systemd](https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html#rpm-running-systemd) or check `/var/log/elasticsearch` and share thsoe logs. Thanks! ---
**Author:** ![Chaim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chaim/32/62552_2.png) [@Chaim](https://discuss.elastic.co/u/Chaim)\ **Post date:** [February 18, 2020, 10:24am UTC](https://discuss.elastic.co/t/e-commerce-logging-strategy/219003/5 "2020-02-18T10:24:33Z")
@spinscale seems like you responded to the wrong post... ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [February 18, 2020, 10:28am UTC](https://discuss.elastic.co/t/e-commerce-logging-strategy/219003/6 "2020-02-18T10:28:51Z")
absolutely! sorry for that. You are right that the rollup integration being improved on currently and needs some more work. You can follow that issue on [https://github.com/elastic/elasticsearch/issues/48003](https://github.com/elastic/elasticsearch/issues/48003) ---
**Author:** ![Chaim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chaim/32/62552_2.png) [@Chaim](https://discuss.elastic.co/u/Chaim)\ **Post date:** [February 18, 2020, 11:49am UTC](https://discuss.elastic.co/t/e-commerce-logging-strategy/219003/7 "2020-02-18T11:49:23Z")
Thanks @spinscale, I saw that post ([https://github.com/elastic/elasticsearch/issues/48003](https://github.com/elastic/elasticsearch/issues/48003)) and also this one ([https://github.com/elastic/elasticsearch/issues/33065](https://github.com/elastic/elasticsearch/issues/33065)) 1. do you have any insight when above may be implemented? I was hoping someone out there has some better solution. On another thought, can we maybe use the transform api? So to concentrate the data in 5 min buckets into a new index then this new index is a regular (non roll-up) index therefore the ILP rollover should work as usual. 1. Does this make sense? Or is my understanding of [transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-transform.html) incorrect... (or maybe some other solution chaining....) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 17, 2020, 11:49am UTC](https://discuss.elastic.co/t/e-commerce-logging-strategy/219003/8 "2020-03-17T11:49:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.