Index management

[tractor_boy]

I am reading through the documentation and cannot find a section that describes how to set up date retention, such that data is deleted as it ages.

I believe its possible to slim data down over time, but i've not yet searched for that.

My scenario currently is that I have data from metric beats going into elastic, and I think its going into one index? I need to clean up the data within this based on retention periods for each item that I have storing.

So cpu may stay for 10 days, disk space 90 days, service state 10 days, etc some of the data may be averaged out before this, and then deletes. What do I need to do to make this happen?

Thanks

Gavin

[AquaX (Andreas Helmer)]

Read this: ILM: Manage the index lifecycle | Elasticsearch Guide [7.15] | Elastic

There are some webinars you can watch as well.

Elasticsearch index lifecycle management: Using the rollover index API

[tractor_boy]

So that is as I understand for the whole index.

My requirement is that I need to manage specific data within the index. Is that possible ootb, or if not has anyone any suggestions on what would be required to make it happen.

Consider a scenario where I collect some data weekly and keep that for a year, but other data I collect every few seconds, but only need that data for a week.

I need to be able to roll up and delete based on the data, not the whole index. Currently I am working with metricbeat, so maybe I have to get metricbeat to write to distinct indexes?

Just need to know how to set thing up correctly before I go to far with setting the system up.

[tractor_boy]

Useful video. I got lost a fair bit when jumping between the ui and running the gets and sets. Is there any reason why all the configuration is not in the ui?

I am still very new to elastic, so am just struggle with the set up and commands, but I'd have zero chance of being able to remember the command actions, whereas the ui is intuitive.

[bhrt]

I guess you could do this:

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "metricbeat-%{[retention]}-%{+yyyy.MM.dd}"

you would have multiple indices created separated by custom retention "tag" added to specific data, and index creation date.
You could then drop all old data based on index name.
I'm also quite new to Elastic so maybe there is a better way to cope with this.