saveToEs Write performance (elasticsearch-spark)

[larghir]

Hi @Pat_Humphreys,

see this answer:

Performance degradation when writing to AWS elasticsearch using elasticsearch-hadoop library

@larghir The situation with keys being shuffled to one reducer is primarily a MapReduce case. A Spark RDD will write out to Elasticsearch in parallel using which ever number of partitions are configured. Writing parallelism does also depend on your RDD layout, your configuration, and the available resources in your environment.

I ended up using saveToEsWithMeta, configuration including at least values for es.batch.size.entries, es.batch.size.bytes, es.batch.write.retry.count.