100% heap usage during indexing

Hey guys,

I have a few questions:

  1. I was wondering if anyone knows how to limit how much memory
    elasticsearch uses while indexing.
    During our loading jobs, typically we only see 60-70% heap usage before the
    garbage collector
    kicks in, but recently we saw some large traffic while loading and the heap
    went to 100%.
    After that everything became unresponsive each time the garbage collector
    would run. As an example
    it spent a full minute running to clear only 46.5mb. Eventually we had to
    restart ES.

We use mostly default settings for ES. We only changed the cluster and
node names, and
changed the index and search threadpools to be of type cached. Is the
unlimited threadpool
for indexing to blame, or are we missing a setting somewhere? I should
mention we are currently
using v1.0.0 ES.

  1. Also, at one point we had a cluster failure (still investigating the
    cause), and after
    rebalancing the speed was very slow, often resulting in timeouts. When the
    cluster returned,
    the speed went back to normal. Is this just an inevitable consequence of
    losing a cluster,
    or should speeds be comparable to before the failure after rebalance?

  2. My lead also wanted me to ask about RAM allocation. We read that you
    should not allocate
    more than 50% of the available RAM to ES because Lucene needs the rest for
    filesystem caching.
    Will this show up as used RAM in top? He said he didn't see the rest of
    the RAM being used.

Thanks in advance guys. We are new to ElasticSearch, and we thought we
would ask some advice
before crashing the site again.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8bcf59bd-c3c1-4eeb-ae33-53d06c33f685%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.