Save a field into another index

Elastic Stack Elasticsearch
1

Hi folks,

I am using the mapper size plugin to save the size of the document to be able to create dashboards, so a developer can see how much log data he produces within his containers in kubernetes/openshift. I am using Filebeat -> Logstash -> Elastic for this setup.

The problem is now that when the retention time for the index is over and the index is deleted, the _size information is also gone. My goal is to keep it for statistics and analytics for longer period of time and the question is:

How can get the _size field along with some other information (like namespace, container name) and save it to another index (lets say sizes_index) with its own ILM policy? I was thinking just to add another output to my logstash pipeline but the _size field is created during the index time in Elasticsearch and is not present for logstash pipelines

Thanks for any hint.

2

Analyze index disk usage API | Elasticsearch Guide [7.16] | Elastic might be an alternative you can use here, as long as you are on a newer version.

However you can probably do this with a rollup - Rolling up historical data | Elasticsearch Guide [7.16] | Elastic

3

Thanks @warkolm for quick response.

Analyze index disk usage API | Elasticsearch Guide [7.16] | Elastic might be an alternative you can use here, as long as you are on a newer version.

This is interesting, but unfortunately not fitting my use case. I need to have the information based on the namespaces and container names which are also part of the documents, so I need to know the size of each individual document and aggregate it based on the namespaces.

However you can probably do this with a rollup - Rolling up historical data | Elasticsearch Guide [7.16] | Elastic

This sounds interesting. I will look into this!