Millions of closed indices?

[cjbottaro (Christopher J. Bottaro)]

Hello. We're toying with the idea of using Es for some type of "cold storage". Basically archival of our clients' data. The data grows indefinitely over time.

The idea is to have an index for each piece of archived data, so over time that could grow to be millions. But we would keep all the indices closed unless one was needed (very rarely). Then we'd open it, do some queries, and then close the index again after some period of inactivity.

Would this be ok to do? Or would it better to pool these indices together into a timeframe? It would be more beneficial to us to have one index per piece of archived data because the schema is a part of what is archived... but we could make pooled indices work.

Thanks for the help.

[Christian_Dahlqvist]

Information about indices and shards still need to be stored in the cluster state even if they are closed so scaling that way will run into bottlenecks way before a million indices/shards. I would therefore recommend pooling indices do you have fewer larger ones.