I am using logstash to process the logs and store the data in an elasticsearch index. I need logs of only a week to be stored and the remaining to be removed from the elasticsearch ( because the index is growing large due to many logs) . Is there anyway we can delete logs(like running a daily job) from elasticsearch index and save the disk space ?
There is a utility called Curator that is designed to do exactly that.
Curator is used at index level. Is there anything at docs level so we can just delete specific docs and new docs will get added to the same existing index
If you are analysing logs, which tends to be immutable, using time based indices, which Curator heips manage, is the most efficient way to deal with data retention. Explicitly deleting records from an index can be done through the delete by query plugin, but is much less efficient than simply deleting indices once the data they hold has exceeded the retention period.