On Wed, Aug 1, 2012 at 10:31 AM, Glyton Camilleri firstname.lastname@example.org wrote:
Each node in the cluster is equipped with 8GB of RAM, running on 512MB of
heap, and handling 6 indices.
Considering the size of the index you mentioned, 512 MB of heap does
seem a bit low. In our installation of ES, we typically allocate
enough heap to keep the entire index in memory, although Shay has
mentioned this is not necessary. I'd suggest you tweak these and see
I noticed that one of the indices becomes much bigger than i would expect:
it reaches a size multiple times larger than the indexed data. The index in
question stores a double type field which the others do not. I set the
precision_step attribute in the mapping for this type in an effort to reduce
the size, also compressing the _source field (which i need present in the
response) and disabling _all; i am also storing and analyzing only what is
Have you tried optimizing your indices?
It often reduces index size and query speed.
Is there any good way to tune the indices and configurations of machines,
and to know what exactly is going on (I use bigdesk to get a handle on
what's going on atm)? And, at the risk of being a bit vague, can such a
problem be solved by configuration, or do i still need to add more nodes to
the cluster anyway in the end?
There are lots of knobs you can tweak to keep your cluster healthy.
What knobs to tweak, or how big your cluster is, is usually a function
of what you're optimizing for. If you're optimizing for indexing
speed, then you'll be looking at your index mapping, and minimize
analysis and storing unnecessary data. If you're optimizing for query
speed, then you're dealing with index structure, query structure, and
a number of application-level knobs I don't know enough about to
comment on at length. If your data is very large, but you don't get
many queries, you may want to have one node with a lot of memory. If
you get lots of queries on a small index, more nodes makes sense.
Happy to be corrected here if I'm talking out of my ass!