Cluster yarn foreachPartition


#1

I run a aplication in cluster yarn, but I have a problem in deploy-mode cluster
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:225)

This is I need do, put documentsinElasticsearch (client = client for elasticsearch)
data.foreachPartition(new VoidFunction<Iterator>() {
public void call(Iterator s) {
while (s.hasNext()) {
client.putDocumentInElasticsearch(s.next());
}
}
});

Client configure
client = new TransportClient(settings).addTransportAddress(new
InetSocketTransportAddress(server, port)

which is the good form for do the conection?

I try diferences ways, but any work

data.foreachPartition(new VoidFunction<Iterator>() {
public void call(Iterator s) {
while (s.hasNext()) {
client=new ElasticSearch()
client.conectToElasticsearch();
client.addDocument(s.next());
client.closeElasticsearch();
}
}
});

Even I try with
EsSpark.saveToEs(rdd,"tweets,tweet");

but It return this error:
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens when accessing a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'

Any suggestions?

Thanks for help.


(system) #2