How many Aliases can an Index have?

[tbart01]

We have an index with some 250,000 aliases, and rapidly growing. Are there any known limits that we should be aware of? We were not able to find any concrete information searching the web/documentation.

Lately, we've noticed that it is taking longer to add new aliases to the index. We have also noticed an instance of "failed to process cluster event (index-aliases) within 30s", but this only happened one time so far.

[warkolm (Mark Walkom)]

Welcome to our community!

@tbart01

We have an index with some 250,000 aliases

@tbart01

Lately, we've noticed that it is taking longer to add new aliases to the index. We have also noticed an instance of "failed to process cluster event (index-aliases) within 30s"

Because each alias addition needs to update the cluster state, and I have never seen of anyone with that many so it's not a surprise it's slow!

Why are you taking this approach?

[tbart01]

@warkolm

Welcome to our community!

Thanks!

@warkolm

Because each alias addition needs to update the cluster state, and I have never seen of anyone with that many so it's not a surprise it's slow!

Thanks for confirming our suspicions.

@warkolm

Why are you taking this approach?

Without going into unnecessary detail, we are using aliases for ease of searching by making use of prefix wildcards on the alias name during search.

Due to the above issues, we are now looking into replacing the alias filters with search-time query filters; we just hope that we do not hit some sort of query limitation, as we will be requiring up to 300 filters to be added onto the query.

[warkolm (Mark Walkom)]

300 is better than 250000, so I would suggest that's a good start.

[Christian_Dahlqvist]

@tbart01

Due to the above issues, we are now looking into replacing the alias filters with search-time query filters; we just hope that we do not hit some sort of query limitation, as we will be requiring up to 300 filters to be added onto the query.

Queries with 300 clauses will also add overhead. It may be useful if you could explain a bit about your data and the problem you are trying to solve using these immense number of aliases and additional query clauses. Maybe someone can provide an alternate and more efficient approach.

[tbart01]

@Christian_Dahlqvist

It may be useful if you could explain a bit about your data and the problem you are trying to solve using these immense number of aliases and additional query clauses. Maybe someone can provide an alternate and more efficient approach.

Here's a very high level of what we have:

We have around 200,000 unique schemas, and 7,000 "users", but not all users have access to all schemas.

Originally, we had an index per schema, but we quickly ran into issues as indices grew. We scrapped everything and decided on a "generic" schema which can handle most (as we have more than a single index to share the load of schemas) schemas.

Since all users are now searching against a handful of indices instead of 200,000, we decided to employ alias filters for a few reasons:

• shorten ES query
• convenient access to a specific user's data

[Christian_Dahlqvist]

What is the minimum, maximum and average number of documents per schema?

What is the minimum, maximum and average number of schemas a user has access to?

[tbart01]

@Christian_Dahlqvist

What is the minimum, maximum and average number of documents per schema?

10, 10 million, 500,000, respectively.

@Christian_Dahlqvist

What is the minimum, maximum and average number of schemas a user has access to?

So, "user" is a placeholder for what is actually out there. We have "clients", and those clients can have tens of thousands of end-users. A single end-user can have access to up to 300 schemas, and thus we would create the query for that specific user. So, the answer to your question is 1, 300, 50, respectively.