Freeze indecies

Elastic Stack Elasticsearch
1

we have an application that create a lot of indices (about 400,000 a week).
there for we're trying to find a solution for the high memory usage this indices requiter.
we are looking at two option ES gives

  1. freeze indices
  2. open/close indices

all of our indices are been indexed in about a day, they are all immutable. we're heavily querying them for about a week, and after that we save them for 90 days in which we can still query them a lot and we need a good query performance for that

what is the difference between Unfreeze index and Open index? which one cast more time a nd performance? which is more suited for our needs? is there a way to freeze or unfreeze by a policy that we create?
also, is there a way to open an index and to close another in an LRU (last recently use) sort of way?

2

That sounds like far too many indices (and shards) for any size cluster. The cluster state is going to be very large and potentially take a long time to update even if you close or freeze them, which will cause all sorts of problems. Have a look at this blog post. Although this blog post is old it may also be applicable.

I would recommend you reconsider your indexing and sharding approach. If you can describe the use case and requirements in greater detail you might be able to get some pointers from the community.

3 Likes
3

Just to echo Christian's comments - that's seriously excessive. You might want to reconsider your strategy.

4

I suspect you will run into issues even if you have virtually empty indices so it should be easy to test.

5

Hi, thanks for the fast response
these are tests logs.. and we do run a lot of tests. also for each test we have ~10 logs, each one has it's own index for it's documents and data is different.

can you explain to me why is that a problem? isn't ES Horizontally scalable? we do get this much indexes created today with our MongoDB configurations..

we do see a potential problem considering the RAM each open index requirements, this is why we considering this freeze/close options.

do you have a take on which approach, freeze/Unfreeze or close/open, is better for our needs? or which one is more costly considering the time of reopen a close index vs unfreeze a freeze index?

6

Elasticsearch is different from MongoDB and can typically not handle that number of indices and shards in a single cluster due to the size of the cluster state. It scales horizontally with respect to data volume but not necessarily index and shard count. I would recommend you create indices by indexing a single document into each and see how many indices you reach before running into problems.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.