Term Index Interval


(phobos182) #1

I have an ES cluster that I am working on setting up. The total index size for all shards will be about 7-12 Billion documents. I'm investigating the trade offs between memory, and search performance. What would be a good setting for term_index_interval for this type of cluster?

I have about 12-15 fields per "type", and about 12 different types. I have tried setting the index interval to 1024, but then queries take seconds to return.

Any advice?

Thanks.


(Shay Banon) #2

You have two types of settings: index.term_index_interval (defaults to 128),
and index.term_index_divisor (defaults to 1). Multiplying the two controls
how many terms are loaded into memory to improve search time. Lower values
provides better search response, at the expense of more memory used.

index.term_index_interval applies when indexing, you can change that
dynamically, but it will only apply to new documents indexed.
index.term_index_divisor can be changed dynamically and applies immediately.

Lower values for index.term_index_interval also control the size of the
index on disk.

Now that we have that out of the way, I think that what you should do is
test..., its hard to give performance recommendations. Sadly, one thing that
would have helped is understanding the amount of memory used for this in
memory term index, but thats not exposed in Lucene (I have a patch somewhere
that tried exposes it, but its quite old).

-shay.banon

On Fri, Sep 2, 2011 at 6:50 PM, phobos182 phobos182@gmail.com wrote:

I have an ES cluster that I am working on setting up. The total index size
for all shards will be about 7-12 Billion documents. I'm investigating the
trade offs between memory, and search performance. What would be a good
setting for term_index_interval for this type of cluster?

I have about 12-15 fields per "type", and about 12 different types. I have
tried setting the index interval to 1024, but then queries take seconds to
return.

Any advice?

Thanks.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Term-Index-Interval-tp3304622p3304622.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(system) #3