# Calling union on two dataframes from spark and elastic search stuck
**URL:** https://discuss.elastic.co/t/calling-union-on-two-dataframes-from-spark-and-elastic-search-stuck/353599
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [February 19, 2024, 1:21pm UTC](https://discuss.elastic.co/t/calling-union-on-two-dataframes-from-spark-and-elastic-search-stuck/353599 "2024-02-19T13:21:46Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@xsa\_xsa](https://discuss.elastic.co/u/xsa_xsa)
#### Post date: [February 19, 2024, 1:21pm UTC](https://discuss.elastic.co/t/calling-union-on-two-dataframes-from-spark-and-elastic-search-stuck/353599/1 "2024-02-19T13:21:46Z")
i am using pyspark to query elstic search
i have the folowing code
d1 = data\_frame1.collect() # return 1 row
d2 = data\_frame2.collect() # return no rows
but when i call
d3 data\_frame1.union(data\_frame2).collect()
the code us stuck
the collection i am quering has 20 rows
the schema of data\_frame1 and data\_frame2 is the same
when i call collect on each it returns correct result
but when i merge them and call collect it stuck
i am using elasticsearch-spark-30\_2.12-8.9.0
Elasticsearch 8.9.2
the query im am using on both data frames is:
'''
{
"bool": {
"must": [{"term": {"field1": "x1213"}}]
}
'''
please help me
---
### Author:  [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [February 20, 2024, 2:05pm UTC](https://discuss.elastic.co/t/calling-union-on-two-dataframes-from-spark-and-elastic-search-stuck/353599/2 "2024-02-20T14:05:46Z")
It seems odd that it would get stuck on such a small amount of data. Have you checked the logs (the logs for the driver as well as the executors running tasks)?
---
### Author:  [@xsa\_xsa](https://discuss.elastic.co/u/xsa_xsa)
#### Post date: [February 21, 2024, 7:01am UTC](https://discuss.elastic.co/t/calling-union-on-two-dataframes-from-spark-and-elastic-search-stuck/353599/3 "2024-02-21T07:01:07Z")
thanks for the reply
i checked the logs of spark and i dont see any errors
its just get stuck on a single task
is there anything i should be looking for?
where do i look for errors?
---
### Author:  [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [February 21, 2024, 1:41pm UTC](https://discuss.elastic.co/t/calling-union-on-two-dataframes-from-spark-and-elastic-search-stuck/353599/4 "2024-02-21T13:41:36Z")
Did you check the task logs on each spark executor? That's where I would guess you would see something, but it is hard to know. You could also check the elasticsearch logs, but I wouldn't think that an elasticsearch error would cause spark to hang.
---
### Author:  [@xsa\_xsa](https://discuss.elastic.co/u/xsa_xsa)
#### Post date: [February 27, 2024, 1:06pm UTC](https://discuss.elastic.co/t/calling-union-on-two-dataframes-from-spark-and-elastic-search-stuck/353599/5 "2024-02-27T13:06:05Z")
ok here is what i found out:
we have 2 indexs
index1- with 20 docs
index2 - with 100000 docs
dataframe1 is a join between 2 dataframes:
dataframe3 - queries index1 (returns 1 row on dataframe3 .collect())
dataframe4 - queries index2 (returns 1 row on dataframe4 .collect())
so
dataframe1 = dataframe3 .join(dataframe4 )
when i call dataframe1.collect() it return 1 row immediatly
dataframe1 return rows from index1
dataframe2 queries index1 with different query (returns 1 row on dataframe2 .collect())
when i do
dataframe1.union(dataframe2).collect() it gets stuck....
what is vert strange is when i don use dataframe4 in the join (so dataframe1 = dataframe3)
everything works fine....
please help
---
### Author:  [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [February 29, 2024, 8:10am UTC](https://discuss.elastic.co/t/calling-union-on-two-dataframes-from-spark-and-elastic-search-stuck/353599/6 "2024-02-29T08:10:30Z")
Can you post your code? Or better yet, could you reproduce it in [this docker image](https://github.com/masseyke/es-spark-docker) and give exact steps to reproduce it?
---
### Author:  [@xsa\_xsa](https://discuss.elastic.co/u/xsa_xsa)
#### Post date: [March 1, 2024, 6:44am UTC](https://discuss.elastic.co/t/calling-union-on-two-dataframes-from-spark-and-elastic-search-stuck/353599/7 "2024-03-01T06:44:23Z")
```auto
# index2 has columns: ["key1", "key2", "key3", "field1"]
# all of the same type
# init data frame 1:
options1 = {
'es.nodes': 'xxxxxxxxxx',
'es.resource': 'index1',
'es.query': '{
"fields": ["key1", "key2", "key3", "key4", "key5", "key6"],
"query": {
"bool": {
"must":[
{
"term": {
"key1": "some_key"
}
}
]
}
},
"_source": False
}'
}
schema1 = ....
reader1 = spark_sesstion
.read.schema(schema1)
.format('org.elasticsearch.spark.sql')
.options(**options1)
df1 = reader1.load().select(["key1", "key2", "key3", "key4", "key5", "key6"])
# init data frame 2:
options2 = {
'es.nodes': 'xxxxxxxxxx',
'es.resource': 'index2',
'es.query': '{
"fields": ["key1", "key2", "key3"],
"query": {
"bool": {
"must":[
{
"term": {
"key1": "some_key",
}
}
]
}
},
"_source": False
}'
}
schema2 = ....
reader2 = spark_sesstion
.read.schema(schema2)
.format('org.elasticsearch.spark.sql')
.options(**options2)
df2 = reader2.load().select(["key1", "key2", "key3"])
df_1_2 = df1.join(df2, ["key1", "key2", "key3"])
# init data frame 3:
options3 = {
'es.nodes': 'xxxxxxxxxx',
'es.resource': 'index1',
'es.query': '{
"fields": ["key1", "key2", "key3", "key4", "key5", "key6"],
"query": {
"bool": {
"must":[
{
"term": {
"some_field: "some field"
}
}
]
}
},
"_source": False
}'
}
reader3 = spark_sesstion
.read.schema(schema1)
.format('org.elasticsearch.spark.sql')
.options(**options3)
df3 = reader3.load().select(["key1", "key2", "key3", "key4", "key5", "key6"])
res = df_1_2.union(df3)
d = res.collect()
```
---
### Author:  [@xsa\_xsa](https://discuss.elastic.co/u/xsa_xsa)
#### Post date: [March 1, 2024, 6:50am UTC](https://discuss.elastic.co/t/calling-union-on-two-dataframes-from-spark-and-elastic-search-stuck/353599/8 "2024-03-01T06:50:39Z")
please please help
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 29, 2024, 6:50am UTC](https://discuss.elastic.co/t/calling-union-on-two-dataframes-from-spark-and-elastic-search-stuck/353599/9 "2024-03-29T06:50:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.