# How to avoid Data too large error for org.apache.spark.sql.streaming.StreamingQueryException
**URL:** https://discuss.elastic.co/t/how-to-avoid-data-too-large-error-for-org-apache-spark-sql-streaming-streamingqueryexception/190604
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [July 15, 2019, 7:44pm UTC](https://discuss.elastic.co/t/how-to-avoid-data-too-large-error-for-org-apache-spark-sql-streaming-streamingqueryexception/190604 "2019-07-15T19:44:32Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@yuecong](https://discuss.elastic.co/u/yuecong)
#### Post date: [July 15, 2019, 7:44pm UTC](https://discuss.elastic.co/t/how-to-avoid-data-too-large-error-for-org-apache-spark-sql-streaming-streamingqueryexception/190604/1 "2019-07-15T19:44:32Z")
I am using spark streaming to dump data from Kafka to ES and I got the following errors.
> org.apache.spark.sql.streaming.StreamingQueryException: Job aborted due to stage failure: Task 6 in stage 4888.0 failed 4 times, most recent failure: Lost task 6.3 in stage 4888.0 (TID 58565, 10.139.64.27, executor 256): org.apache.spark.util.TaskCompletionListenerException: org.elasticsearch.hadoop.rest.EsHadoopRemoteException: circuit\_breaking\_exception: [parent] Data too large, data for [\] would be [6104859024/5.6gb], which is larger than the limit of [6103767449/5.6gb], real usage: [6104859024/5.6gb], new bytes reserved: [0/0b]
* * *
Could someone suggest how I can adjust some parameters to avoid this?
Here is my es configurations to write to ES.
val esURL = "xxxx"
serviceLogDfForES.writeStream
.outputMode("append")
.format("org.elasticsearch.spark.sql")
.option("es.nodes.wan.only","true")
.option("es.port","9200")
.option("es.net.http.auth.user", "xxx")
.option("es.net.http.auth.pass", "xxx")
.option("checkpointLocation", "/mnt/xxxx/\_checkpoint1")
.option("es.net.ssl","true")
.option("es.net.ssl.cert.allow.self.signed", "true")
.option("es.mapping.date.rich", "true")
.option("es.nodes", esURL)
.option("es.resource.write", "service-log-{date}")
.start().awaitTermination()
---
### Author:  [@yuecong](https://discuss.elastic.co/u/yuecong)
#### Post date: [July 15, 2019, 8:10pm UTC](https://discuss.elastic.co/t/how-to-avoid-data-too-large-error-for-org-apache-spark-sql-streaming-streamingqueryexception/190604/2 "2019-07-15T20:10:02Z")
I checked my cluster, and it works fine
{
"cluster\_name" : "logs001",
"status" : "green",
"timed\_out" : false,
"number\_of\_nodes" : 8,
"number\_of\_data\_nodes" : 3,
"active\_primary\_shards" : 14,
"active\_shards" : 29,
"relocating\_shards" : 0,
"initializing\_shards" : 0,
"unassigned\_shards" : 0,
"delayed\_unassigned\_shards" : 0,
"number\_of\_pending\_tasks" : 0,
"number\_of\_in\_flight\_fetch" : 0,
"task\_max\_waiting\_in\_queue\_millis" : 0,
"active\_shards\_percent\_as\_number" : 100.0
}
Also if I rerun the spark streaming job, it started to work.
---
### Author:  [@yuecong](https://discuss.elastic.co/u/yuecong)
#### Post date: [July 17, 2019, 5:57am UTC](https://discuss.elastic.co/t/how-to-avoid-data-too-large-error-for-org-apache-spark-sql-streaming-streamingqueryexception/190604/3 "2019-07-17T05:57:36Z")
@james.baiera do you have some insights on this? Thanks
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 14, 2019, 5:57am UTC](https://discuss.elastic.co/t/how-to-avoid-data-too-large-error-for-org-apache-spark-sql-streaming-streamingqueryexception/190604/4 "2019-08-14T05:57:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.