# Caused by: org.apache.spark.SparkException: Data of type java.util.GregorianCalendar cannot be used **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [August 8, 2019, 4:33pm UTC](https://discuss.elastic.co/t/caused-by-org-apache-spark-sparkexception-data-of-type-java-util-gregoriancalendar-cannot-be-used/194471 "2019-08-08T16:33:51Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Mayukh](https://avatars.discourse-cdn.com/v4/letter/m/ecd19e/32.png) [@Mayukh](https://discuss.elastic.co/u/Mayukh)\ **Post date:** [August 8, 2019, 4:33pm UTC](https://discuss.elastic.co/t/caused-by-org-apache-spark-sparkexception-data-of-type-java-util-gregoriancalendar-cannot-be-used/194471/1 "2019-08-08T16:33:51Z")
Hi, I am trying to write a PySpark RDD into Elasticsearch, however I am getting the GregorianCalendar error. Any suggestions or help would be appreciated. ``` es_fault_df = sqlContext.sql("select * from fault_search_all limit 10") es_fault_rdd = es_fault_df.rdd.map(lambda item: ('key', {'objid': item['objid'],'customer_name': item['customer_name'] ,'veh_hdr':item['veh_hdr'],'road_number':item['road_number'],'fleet_name':item['fleet_name'],'model_desc':item['model_desc'],'fault_code':item['fault_code'],'sub_id':item['sub_id'],'fault_desc':item['fault_desc'],'occur_date':item['occur_date'],'fault_reset_date':item['fault_reset_date'],'fault_origin':item['fault_origin'],'record_type':item['record_type'],'gps_latitude':item['gps_latitude'],'gps_longitude':item['gps_longitude'],'offboard_load_date':item['offboard_load_date'],'loco_speed':item['loco_speed'],'engine_speed':item['engine_speed'],'notch':item['notch'],'direction':item['direction'],'hp':item['hp'],'water_temp':item['water_temp'],'oil_temp':item['oil_temp'],'mode_call':item['mode_call'],'loco_state_desc':item['loco_state_desc'],'software_subid':item['software_subid']})) es_fault_rdd.saveAsNewAPIHadoopFile(path='-',outputFormatClass="org.elasticsearch.hadoop.mr.EsOutputFormat",keyClass="org.apache.hadoop.io.NullWritable", valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable", conf=es_conf) ``` Logs: ``` Caused by: org.apache.spark.SparkException: Task failed while writing rows at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.org$apache$spark$internal$io$SparkHadoopMapReduceWriter$$executeTask(SparkHadoopMapReduceWriter.scala:178) at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$3.apply(SparkHadoopMapReduceWriter.scala:89) at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$3.apply(SparkHadoopMapReduceWriter.scala:88) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ... 1 more Caused by: org.apache.spark.SparkException: Data of type java.util.GregorianCalendar cannot be used at org.apache.spark.api.python.JavaToWritableConverter.org$apache$spark$api$python$JavaToWritableConverter$$convertToWritable(PythonHadoopUtil.scala:141) at org.apache.spark.api.python.JavaToWritableConverter$$anonfun$org$apache$spark$api$python$JavaToWritableConverter$$convertToWritable$1.apply(PythonHadoopUtil.scala:134) at org.apache.spark.api.python.JavaToWritableConverter$$anonfun$org$apache$spark$api$python$JavaToWritableConverter$$convertToWritable$1.apply(PythonHadoopUtil.scala:133) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at org.apache.spark.api.python.JavaToWritableConverter.org$apache$spark$api$python$JavaToWritableConverter$$convertToWritable(PythonHadoopUtil.scala:133) at org.apache.spark.api.python.JavaToWritableConverter.convert(PythonHadoopUtil.scala:148) at org.apache.spark.api.python.JavaToWritableConverter.convert(PythonHadoopUtil.scala:115) at org.apache.spark.api.python.PythonHadoopUtil$$anonfun$convertRDD$1.apply(PythonHadoopUtil.scala:181) at org.apache.spark.api.python.PythonHadoopUtil$$anonfun$convertRDD$1.apply(PythonHadoopUtil.scala:181) at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$4.apply(SparkHadoopMapReduceWriter.scala:147) at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$4.apply(SparkHadoopMapReduceWriter.scala:144) at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1371) at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.org$apache$spark$internal$io$SparkHadoopMapReduceWriter$$executeTask(SparkHadoopMapReduceWriter.scala:159) ... 8 more ``` ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [August 14, 2019, 7:25pm UTC](https://discuss.elastic.co/t/caused-by-org-apache-spark-sparkexception-data-of-type-java-util-gregoriancalendar-cannot-be-used/194471/2 "2019-08-14T19:25:51Z")
This doesn't seem like an ES-Hadoop specific problem but more of a Spark issue? I would check with the Spark mailing list about this, but more likely, it's possible that Spark just does not support that data type. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 11, 2019, 7:25pm UTC](https://discuss.elastic.co/t/caused-by-org-apache-spark-sparkexception-data-of-type-java-util-gregoriancalendar-cannot-be-used/194471/3 "2019-09-11T19:25:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.