Basically we want to be able to index a document from Logstash using the ‘document_id’ parameter in the Elasticsearch output to overwrite\update a document and also use curator to delete anything older than 30days.
We have 2 options for this:
- Have a single index so the document_id parameter will function in our requirements were it overwrites\updates the existing document but the downfall of this is being able to delete indices older than 30days.
- Have a day base index to allow us to delete data older than 30 days but the document_id will only function if it is called on the same day. If a document is indexed the day after we get 2 documents with the same _id, one in todays and one in yesterdays.
Is there a known work around for this