I am using the mapper size plugin to save the size of the document to be able to create dashboards, so a developer can see how much log data he produces within his containers in kubernetes/openshift. I am using Filebeat -> Logstash -> Elastic for this setup.
The problem is now that when the retention time for the index is over and the index is deleted, the
_size information is also gone. My goal is to keep it for statistics and analytics for longer period of time and the question is:
How can get the
_size field along with some other information (like namespace, container name) and save it to another index (lets say
sizes_index) with its own ILM policy? I was thinking just to add another output to my logstash pipeline but the
_size field is created during the index time in Elasticsearch and is not present for logstash pipelines
Thanks for any hint.