# Spark job hangs on the union() operation

**URL:** https://discuss.elastic.co/t/spark-job-hangs-on-the-union-operation/56728
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [July 29, 2016, 12:43pm UTC](https://discuss.elastic.co/t/spark-job-hangs-on-the-union-operation/56728 "2016-07-29T12:43:37Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![aokolnychyi](https://avatars.discourse-cdn.com/v4/letter/a/f05b48/32.png) [@aokolnychyi](https://discuss.elastic.co/u/aokolnychyi)
#### Post date: [July 29, 2016, 12:43pm UTC](https://discuss.elastic.co/t/spark-job-hangs-on-the-union-operation/56728/1 "2016-07-29T12:43:37Z")

</div>

Hi,

my Spark Job hangs while I am trying to compute a union of 2 RDDs after the aggregateByKey operation.

I have the following index:  
`{ "test_index": { "aliases": {}, "mappings": { "keyValue": { "properties": { "key": { "type": "string", "index": "not_analyzed" }, "value": { "type": "double" } } } }, "settings": { "index": { "creation_date": "1469795011108", "number_of_shards": "5", "number_of_replicas": "1", "uuid": "6HIbbHcLSv6pctgbh1mt9A", "version": { "created": "2030299" } } }, "warmers": {} } }`

and the following code that hangs on computing a union of 2 RDDs **_ONLY_** after the aggregationByKey operation:

`val configuration = new SparkConf()  
.setAppName("ES Spark Test Application")  
.setMaster("local[4]")

val sparkContext = new SparkContext(configuration)

val firstEsRDD = sparkContext.esRDD("test\_index/keyValue")  
.map { case (id, data) =\> (data("key").asInstanceOf[String], 1) }  
val secondEsRDD = sparkContext.esRDD("test\_index/keyValue")  
.map { case (id, data) =\> (data("key").asInstanceOf[String], 1) }

secondEsRDD.union(firstEsRDD).collect().foreach(println) **// works**  
secondEsRDD.aggregateByKey(0)(\_ + \_, \_ + _).collect().foreach(println) **// works**  
secondEsRDD.aggregateByKey(0)(_ + \_, \_ + \_).union(firstEsRDD).collect().foreach(println) **// hangs**

val firstRegularRDD = sparkContext.parallelize(Array(("a", 1), ("a", 2), ("b", 2)))  
val secondRegularRDD = sparkContext.parallelize(Array(("a", 1), ("a", 2), ("b", 2)))  
firstRegularRDD.union(secondRegularRDD).collect().foreach(println) **// works**  
firstRegularRDD.aggregateByKey(0)(\_ + \_, \_ + _).collect().foreach(println) **// works**  
firstRegularRDD.aggregateByKey(0)(_ + \_, \_ + \_).union(secondRegularRDD).collect().foreach(println) **// works** `

Spark version is 1.6.1. ES-Hadoop version is 2.3.1.

Any suggestions on what I am doing wrong are more than welcome.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:23pm UTC](https://discuss.elastic.co/t/spark-job-hangs-on-the-union-operation/56728/2 "2017-07-06T13:23:44Z")

</div>