Data stream backing indices don't support aliases

[jonnymccullagh]

First of all let me admit that I am a newbie who has not found an explanation of how filebeat/metricbeat, data streams, indices and index templates work/flow together. I am using beats to send logs, metrics and traces into elastic cloud. I have found that my running of metricbeat has created a hidden index named .ds-metricbeat-8.4.2-2022.12.06-000001 and similar for traces/logs/k8s.
These indices are very large and I want to delete old data, so I created a custom lifecycle policy to delete after 14 days.
I removed the managed lifecycle policy from .ds-metricbeat-8.4.2-2022.12.06-000001 but now I cannot add my custom policy - the UI tells me the index does not have an alias. I tried to add a 'metricbeat' alias to it but:

The provided expressions [.ds-metricbeat-8.4.2-2022.12.06-000001] match a backing index belonging to data stream [metricbeat-8.4.2]. Data stream backing indices don't support aliases

I reckon I have done something wrong in the whole setup as I would have expected a metricbeat index that would have indices behind it that can rollover rather than grow forever.
So my questions are:

1. Should I be telling metricbeat which index to send data into?
2. Should I be installing metricbeat index templates so that I am not using a hidden index?
3. Do I need to delete these hidden indices and start over?

[stephenb (Stephen Brown)]

Hi @jonnymccullagh

In my opinion You should stick with data streams especially for time series data . Did you go and read the docs on them?

There are a number of reasons that we moved to data streams, particularly for time series data, trying to go back to normal indices... you will be "Swimming UpStream" ... pun intended.

(Fixed) Just FYI the data stream name IS the alias for data streams and the backing indices are governed by the the Rollover setting in the ILM policy

Perhaps I'm missing something. Why don't you just go into the existing Metricbeat ILM policy and just set it to what you want. We provide a base or default ILM policy, but it certainly intended to be modified to your needs.

Stack Management -> ILM Policy > Metricbeat

Enable Delete

Then Pick the Time Frame you want to Delete and Save... It should go in and clean up the Old Indices (in a bit of time, it is not instantaneous)

And that Delete is "After the Rollover" So if the index rolls over each day it will then wait 7 additional days for the delete so ~8 days total ..

Perhaps I am missing something ... There are other details etc... that could affect what you want to do...

[jonnymccullagh]

Apologies Stephen you seem to be getting all my dumb questions. I think I was put off by it being a hidden index and the message at the top

You are editing an existing policy. Any changes you make will affect 3 linked indices(opens in a new tab or window) and 1 linked index template that are attached to this policy. Alternatively, you can save these changes in a new policy.

That guided me towards creating my own custom policy and then got into diffs because there was no alias.
We are trying to keep things small until we work out what we need to keep long-term.
I have made the changes you suggested so many thanks. I could probably do with an hour or two with a customer success person.

[stephenb (Stephen Brown)]

No Worries...

You can also adjust these... but there are reasons they are set they way they are ... 50GB / shard at scale is a good number 30 Days so if you have small volume daily ...

BTW I said the Data Streams roll over daily they do not.. that was a typo... they respect the Rollover Setting (Max Shard Size / Max Age) in the ILM Policy

So here is example my POC policy .. 10GB or Daily ... Also at some point you may want to learn about force merge etc

@jonnymccullagh

I could probably do with an hour or two with a customer success person.

Do you have a commercial subscription?

There is lots of free training and webinars...

There is formal Paid Training...

And of course what I think is really the most effective is spend a couple days with our consultants ... you get the job done, best practices, tuned for your use case and you Learn a Bunch all at the same time.

That is my experience... from seeing lots of implementations...