How many months of log data does organization should store in Elasticsearch? Do we store the log data anywhere aside from Elasticsearch (e.g. flat files?)
How do we archive old log data that is required for regulatory compliance but not needed “online” in the Elasticsearch cluster?
The usual answer here is 'it depends'. First, the amount of data to retain depends on your budget, your hardware and potentially also in legal requirements of the country where the data is stored or processed. Especially sometimes in the latter case you dont have a choice.
Regarding archiving, you might want to check out Index Lifecycle Management and soon snapshot lifecycle management (you should take a look at snapshots as well). Also frozen indices are helpful in this case.
1 Like
Thanks for your answer,
actually I am looking to find out what's best practices of other company's.
Now I am throwing another question ?
- Can we have SOAP logs retention for week and others for more time as long we needed.
regarding your other question: easiest would be to store your SOAP logs in a dedicated index and use ILM to automatically delete those indices (or do it manually) after a different time period compared to the other indices.
Dear Spinscale,
can you attach web link of ILM, which you have mentioned for reply of my other question.
is there other tools available beside curator for archiving data or not ?
when we archive the data or back up what's best format to compress the data or do we have some tools to encrypt them as well or not ?
ILM refers to Index Lifecycle Management, which I linked in the above post already - sorry for the confusion.
is there other tools available beside curator for archiving data or not ?
when we archive the data or back up what's best format to compress the data or do we have some tools to encrypt them as well or not ?
Check out elasticdump:
1 Like
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.