# Data of type java.util.GregorianCalendar cannot be used

**URL:** <https://discuss.elastic.co/t/data-of-type-java-util-gregoriancalendar-cannot-be-used/93573>\
**Category:** Elasticsearch\
**Created:** [July 18, 2017, 11:56am UTC](https://discuss.elastic.co/t/data-of-type-java-util-gregoriancalendar-cannot-be-used/93573 "2017-07-18T11:56:41Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![SailLiao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sailliao/32/20171_2.png) [@SailLiao](https://discuss.elastic.co/u/SailLiao)\
**Post date:** [July 18, 2017, 11:56am UTC](https://discuss.elastic.co/t/data-of-type-java-util-gregoriancalendar-cannot-be-used/93573/1 "2017-07-18T11:56:41Z")

</div>

i use pyspark read a parquet file from hdfs and i hive a column type is timstamp , when i write the DataFrame to es , there is an exception **Data of type java.util.GregorianCalendar cannot be used** but when i not use the timestamp column , it works fine.

```
spark = SparkSession.builder.appName(" python es ") .config("spark.master", "yarn") .getOrCreate()
path = "/data/file/test.parquet"
df = spark.read.load(path, format='parquet')
esClient = ESClient(index=db_name, type=table_name, id_field=id_filed)
esClient.writeDF2ES(df)

es_conf={
      'es.mapping.id': 'xxxxx'
      'es.nodes': 'xxxxx'
      'es.resource': 'xxxxx'
}

def writeRDD2ES(rdd):
    try:
        rdd.saveAsNewAPIHadoopFile(
            path='-',
            outputFormatClass="org.elasticsearch.hadoop.mr.EsOutputFormat",
            keyClass="org.apache.hadoop.io.NullWritable",
            valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",
            conf=es_conf)
    except:
        self.logger.error("Failed to write es(%s)" % self)
        raise

```

* * *

Caused by: org.apache.spark.SparkException: Data of type java.util.GregorianCalendar cannot be used  
at org.apache.spark.api.python.JavaToWritableConverter.org$apache$spark$api$python$JavaToWritableConverter$$convertToWritable(PythonHadoopUtil.scala:141)  
at org.apache.spark.api.python.JavaToWritableConverter$$anonfun$org$apache$spark$api$python$JavaToWritableConverter$$convertToWritable$1.apply(PythonHadoopUtil.scala:134)  
at org.apache.spark.api.python.JavaToWritableConverter$$anonfun$org$apache$spark$api$python$JavaToWritableConverter$$convertToWritable$1.apply(PythonHadoopUtil.scala:133)  
at scala.collection.Iterator$class.foreach(Iterator.scala:893)  
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)  
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)  
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)  
at org.apache.spark.api.python.JavaToWritableConverter.org$apache$spark$api$python$JavaToWritableConverter$$convertToWritable(PythonHadoopUtil.scala:133)  
at org.apache.spark.api.python.JavaToWritableConverter.convert(PythonHadoopUtil.scala:148)  
at org.apache.spark.api.python.JavaToWritableConverter.convert(PythonHadoopUtil.scala:115)  
at org.apache.spark.api.python.PythonHadoopUtil$$anonfun$convertRDD$1.apply(PythonHadoopUtil.scala:181)  
at org.apache.spark.api.python.PythonHadoopUtil$$anonfun$convertRDD$1.apply(PythonHadoopUtil.scala:181)  
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)  
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1124)  
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)  
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)  
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1341)  
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)  
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)  
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)  
at org.apache.spark.scheduler.Task.run(Task.scala:99)  
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 15, 2017, 11:56am UTC](https://discuss.elastic.co/t/data-of-type-java-util-gregoriancalendar-cannot-be-used/93573/2 "2017-08-15T11:56:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.