I would actually recommend against this configuration in particular at least on the es-hadoop side.
When a node keeps on dropping one should try minimizing the load and see whether it has any impact - the configuration
below does the opposite:
- increases the batch size to 300 million or around 286mb (by the way, use a unit to simplify the configuration and make
it human readable - 285mb).
- increases the batch number of entries to 10K
Note the above represent the upper limit for a batch for each Spark task; that is at runtime, one has to multiply the
number of tasks with the batch size to
get the maximum number of batching done when all the tasks at running; in this case it can be around 2850mb or more than
2.5 gb at once against the cluster which is excessive.
Bulk should be significantly smaller to allow ES to quickly ingest it and move to the next. As explained in the docs, a
bulk request should take 1-2s max otherwise
either increase the hardware or minimize the bulk size.
Increasing the wait time/timeout and number of retries is also counter productive since in case of an error, it simply
takes longer for the client to react. These settings
should be used when the network is throttled or to go around potential spikes in the system. However hammering the
system (with bulk size way too big) and working around the rejections
by increasing the wait time and the number of retries is not solving your system performance but rather aggravating and
postponing the issue.
Hope this helps,
On 1/7/15 9:50 AM, sushmitha wrote:
Hi,
I got the similar exception when I tried to index the data. I resolved this
by configuring "elasticsearch.yml" as
bootstrap.mlockall: true
index.number_of_shards: 40
index.term_index_interval: 1000
index.term_index_divisor: 5
index.translog.flush_threshold_period: 5s
index.refresh_interval: 5s
index.merge.policy.use_compound_file: false
index.compound_format: false
indices.memory.index_buffer_size: 40%
And add this configuration in your code when you initialize sparkcontext
object.
sparkConf.set("es.batch.size.bytes", "300000000")
sparkConf.set("es.batch.size.entries", "10000")
sparkConf.set("es.batch.write.refresh", "false")
sparkConf.set("es.batch.write.retry.count", "50")
sparkConf.set("es.batch.write.retry.wait", "500")
sparkConf.set("es.http.timeout", "5m")
sparkConf.set("es.http.retries", "50")
sparkConf.set("es.action.heart.beat.lead", "50")
Apart from these, set ES_HEAP_SIZE also. I think setting these things will
resolve your problem
Thanks,
Sushmitha
--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Elasticsearch-Spark-EsHadoopNoNodesLeftException-in-cluster-Mode-tp4068066p4068603.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.
--
Costin
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54AD0763.7070702%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
