Load data into HDFS using ES-Spark

Elastic Stack Elasticsearch
1

Hello,

I am reading data from elasticsearch using spark (ES-Spark). After i get
the data using sc.esRDD(".../...") I want to store everything in HDFS so I
use the saveAsTextFile method but it is very slow ...
Am I doing the right things ? It takes 15min to save (and it is saving 11Go)

Best regards

--
Please update your bookmarks! We moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3fb6d71e-ec3d-4ecf-a293-a1ffb23cd04a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hi,

I replied over at Discuss [1].

Thanks,

[1] Load data into HDFS using ES-Spark - #2 by costin
On 5/6/15 5:22 PM, Lucas Weissert wrote:

Hello,

I am reading data from elasticsearch using spark (ES-Spark). After i get the data using sc.esRDD(".../...") I want to
store everything in HDFS so I use the saveAsTextFile method but it is very slow ...
Am I doing the right things ? It takes 15min to save (and it is saving 11Go)

Best regards

--
Please update your bookmarks! We moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3fb6d71e-ec3d-4ecf-a293-a1ffb23cd04a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3fb6d71e-ec3d-4ecf-a293-a1ffb23cd04a%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5554476D.1070005%40gmail.com.
For more options, visit https://groups.google.com/d/optout.