[idea] Would creating "indexes" as "aliases" by default make sense?

Elastic Stack Elasticsearch
1

As a follow up to my post on index renaming I had this idea.

What if Elasticsearch created each index as an alias to the actual index in the first place?

As in, when creating a new index, unless the user overrides it, the index is created as an alias at "index name", with an actual data index at "index name-000001", or maybe ".data.index name".

What my idea is trying to solve is:

  • An index getting so big you can't work with it due to lack of disk space.
  • Misconfiguration creating an index instead of an alias and rollover index. Then hitting the disk space issue.
  • Debug logs filling up your space, and it not being easy to recover the space after you delete said logs from your indexes. (Since deletion doesn't actually recover space until a force merge (I think)? Which might, or might not, work if you've already set replicas to 0 and shards to 1.)
  • Needing to rename an index for any reason.

Using an alias instead of an index could help fix a lot of those issues.

  • Large indices could be more easily added to ILM with rollover if they don't have it set up already.
  • The misconfiguration would just involve fixing the rollover/ilm stuff.
  • To recover deleted docs disk space, you could change the alias name, then recreate the old alias/index and reindex into it. When finished, just delete the old data index and you'd have your disk space back.
  • Renaming would just be creating a new alias and removing the old.

I'm just floating the idea. No idea if it's a good one or not. Nor do I have any idea how it'd get implemented. I'd think I'd have run across anyone floating the same idea while trying to solve the issues I just listed, so I haven't actually looked for any existing posts on this idea. Again, just brainstorming, not trying to get anything done, or make more work for anyone.

Anyone have any thoughts on the idea?

2 Likes
2

Anyone have any thoughts?

1 Like
3

I think this makes WAY more sense than the current index->alias relationship.

It would avoid the whole convoluted bootstrapping process, and allow for dynamic index creation.
You could send an event matching a Template, and it could just create the index(alias), write index, and rollover, without making that a manual process prone to misconfiguration.

This functionality is similar to how Data Streams work, but those are gated by timestamps.
Right now I'm solving this problem with an ingest pipeline adding timestamps where they don't exist, but it feels like this should be the default behavior.

1 Like