Working with logs

[duck]

We're building a logging system for our project. We want to see almost everything as a log like Docker, application console etc. Since we're growing up, we do not want to make mistake about logs(wrong scheme, indexing, querying etc.)

There are few questions in my mind,

1. After some times, log file size will grow up and be a problem. What should we do about it?
2. We want to log user actions like when they purchase something or their message logs etc. For that purpose, what should we do? Should we save their message logs with timestamp to Elasicearch?
3. We want to analyze everything. Should we use timestamp for every document?

I'd like to hear your ideas. Thanks!

[warkolm (Mark Walkom)]

1. Use [ILM]ILM: Manage the index lifecycle | Elasticsearch Guide [8.11] | Elastic).
2. Yes, that's a log
3. If it's time based data then 100% you should

@duck

Since we're growing up, we do not want to make mistake about logs(wrong scheme, indexing, querying etc.)

If we're being honest, you will make a mistake. There's no way to make something completely, 100% correct. You're better off making your system and process resilient to changes.