E-commerce logging strategy

Elastic Stack Elasticsearch
1

Our Current Setup

  • We have an elastic cloud deployment, collecting browsing logs for an e-commerce website, on a regular day our log reaches ~100GB

  • Given the huge amount of data, we only keep 3 days of logs then archive it.

  • We also set up a rollup index as 5 min intervals, grouped by url, status-code, RTT etc.
    the rollup log results in ~5GB data daily

  • We would like to keep the rollup logs for at least 2 years

  • The problem is that after amount its over 100GB and did not find yet a way to automatically rollover...

  • So far we do a new rollup setup monthly. so we get it in a new log.

My question is
Does our setup make sense?

  1. Can you offer some advice on doing things better
  2. Is there any reference / example of a good ecomm logging strategy
2

Hey,

have you seen the rollover API - you might want to take a look at index lifecycle management and frozen indices as well in order to reduce memory footprint.

Hope this helps as a start.

--Alex

3

Thanks @spinscale!

sure, we're actually using the rollover API for our regular indexes, thru the "index lifecycle management".

the challenge we have is combining the 2...
as far as we see, the "rollup_index": "" (in the rollup definition) needs to be set to a static index name (no variable or alias, we tried either and it failed)..

so again to clarify, we love the ILP idea, and want to extand this "set and forget" concept also to our rollup's but we are stuck here...
does my question make more sense now?

4

These logs are not enough. Please see https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html#rpm-running-systemd or check /var/log/elasticsearch and share thsoe logs.

Thanks!

5

@spinscale seems like you responded to the wrong post...

6

absolutely! sorry for that. You are right that the rollup integration being improved on currently and needs some more work. You can follow that issue on https://github.com/elastic/elasticsearch/issues/48003

7

Thanks @spinscale,

I saw that post (https://github.com/elastic/elasticsearch/issues/48003)
and also this one (https://github.com/elastic/elasticsearch/issues/33065)

  1. do you have any insight when above may be implemented?

I was hoping someone out there has some better solution.
On another thought, can we maybe use the transform api? So to concentrate the data in 5 min buckets into a new index then this new index is a regular (non roll-up) index therefore the ILP rollover should work as usual.

  1. Does this make sense? Or is my understanding of transform incorrect... (or maybe some other solution chaining....)
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.