Index Max Size


(Juan Andres Ramirez) #1

Hello Guys ,
Sorry for my noob question, but I can't find where can I setup the index size?. I just see in docs about shard cache or buffer size.

Thank you


#2

Hi! There are no index size limit. Index max size limited by available hardware (in my case with RAM) and some shard-level limits (2.1B documents per shard and recommended shard size up to 20-40GB (soft limit to ensure reasonably fast index relocation speed over the net)).


(Juan Andres Ramirez) #3

Hello Rusty,
Thank you for you answer, with curator I can handle the index. I'll try with it.

thank you !!!!


#4

That means that my index size is limited by the shards and would be amount of shards * max shard size (40 GB or what is it?). When the index size gets close to that I would have to add a shard to the index, correct? Is that possible or do I have to create a new index with let's say double the shards and move the old index to the new one ?


#5

The only limit is 2.1Billion documents per shard (it's Lucene core limitation), 20-40GBytes is soft limit (recommended size in terms of shard relocation time and shard recovery time).

You should plan expected capacity in advance or use for sample time/name based indexes (you_index-DATETIME_OR_COUNTER_OR_UID_OR_MISC and use wildcard to search like "/you_index-*/_search").


#6

True, 2.1 Billion documents was the restriction. Thanks .


(system) #7