Hi, I have ~1TB of data in S3 which I am doing some transformations on using PySpark and then trying to write the result to Elasticsearch. I came across using the ES-Hadoop package however, had to set of es.batch.write.retry.count to -1 to be able to keep writing data successfully. The documentatio…

Indexing data in bulk in Elasticsearch using PySpark

Topic		Replies	Views
Performance of Spark bulk index to Elasticsearch Elasticsearch es-hadoop	3	2666	September 1, 2017
org.elasticsearch.hadoop.EsHadoopException: Could not write all entries for bulk operation [1/1]. Error sample (first [5] error messages): Elasticsearch	1	277	April 8, 2024
[SPARK] es.batch.write.retry.count negative value is ignored Elasticsearch es-hadoop	7	4907	July 6, 2017
How to handle data that causes failure while indexing from spark to ES Elasticsearch es-hadoop	2	2067	October 10, 2017
Is it possible to perform bulk insert from Spark to ElasticSearch? Elasticsearch es-hadoop	4	6579	July 6, 2017