I am ingesting logs which has very high verbosity, in order to cope with the high number of documents I wanted to create a policy in the index lifecyle, rolling over to cold an index after certain number of documents or size is fine, but what I want is that only the documents that are DEBUG or TRACE as they are the 90% of the logs, they roll to be in the phase of delete and keep the rest WARN, FATAL etc for longer.
I could just remove the high verbosity but I need to keep the high verbosity for at least 1 week at all times.
Is there a way to do a query in the policy?
You need to split the different log types into different indices as retention is managed at the index level.
Oh true! I didn't think it like that! I can split the pipeline in logstash for that.
Thank you!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.