It depends on the doc sizes and on the number of nodes that are processing
the bulk requests.
Consider reasonable bulk response times. ES should answer around 1-10
seconds, for that, it should not receive bulk requests > 10mb per node.
Just moving 10mb up to a node will take around a second.
So it is important to streamline bulk indexing, to get reasonable bulk
sizes, and to get reasonable response times.
There are also memory options per node to modify the default bulk indexing
resources per node and per shard.
While you can ramp up many dozens or even hundreds of GB on the heap just
for bulk indexing, it is not efficient to allocate such vast amounts of
received raw bulk data in ES. This will either run into thread pool / queue
limits, or time out if it is transported to replica or to other shards, or
you will put much too high pressure on GC as you have observed.
Jörg
On Thu, Dec 12, 2013 at 2:51 AM, Daniel Guo daniel5hbs@gmail.com wrote:
I use the BulkRequestBuilder to index about 300,000 documents at once.
The ES_HEAP_SIZE is set to 2g. I get the following error:
OutOfMemoryError: GC overhead limit exceed
https://lh4.googleusercontent.com/-gPuV90FoSbY/UqkUssvgllI/AAAAAAAAAVU/XxLEZ_VUP_0/s1600/outOfMemeory.jpg
Except for dividing the bulk request into several smaller bulk requests,
any other suggestions or optimizations?
Thanks a lot.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bf34f259-f21e-472d-a5ab-7122da5182a4%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFuR6JjG-0r1Kcwmm6G%2Bqn6oGzsz0kU-%2B8kFsKx6VYcog%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
