# Elasticsearch 2.0 and Spark - TimestampType conversion issue

**URL:** <https://discuss.elastic.co/t/elasticsearch-2-0-and-spark-timestamptype-conversion-issue/34136>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [November 9, 2015, 10:43am UTC](https://discuss.elastic.co/t/elasticsearch-2-0-and-spark-timestamptype-conversion-issue/34136 "2015-11-09T10:43:53Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![eliasah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eliasah/32/34741_2.png) [@eliasah](https://discuss.elastic.co/u/eliasah)\
**Post date:** [November 9, 2015, 10:43am UTC](https://discuss.elastic.co/t/elasticsearch-2-0-and-spark-timestamptype-conversion-issue/34136/1 "2015-11-09T10:43:53Z")

</div>

I'm trying to write a DataFrame into Elasticsearch 2.0 with the following schema

```
|-- actionId: long (nullable = true)
|-- userId: long (nullable = true)
|-- saleDate: timestamp (nullable = true)

```

When the index is created during the job the `saleDate` fields seems to be converted into a long. Here is a part of the mapping :

```
"saleDate": { "type": "long" },

```

Is this behavior expected? If so How would it be possible to write a time-stamp field without declaring the mapping before-hand?

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [November 9, 2015, 12:47pm UTC](https://discuss.elastic.co/t/elasticsearch-2-0-and-spark-timestamptype-conversion-issue/34136/2 "2015-11-09T12:47:43Z")

</div>

There's no functionality in ES-Hadoop to force a certain type conversion outside the existing type and that is on purpose.  
Elasticsearch can do that much more reliably and better than the connector by declaring the mapping (sometimes templates help a lot) apriori.

---

<div class="post-metadata">

**Author:** ![eliasah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eliasah/32/34741_2.png) [@eliasah](https://discuss.elastic.co/u/eliasah)\
**Post date:** [November 9, 2015, 2:51pm UTC](https://discuss.elastic.co/t/elasticsearch-2-0-and-spark-timestamptype-conversion-issue/34136/3 "2015-11-09T14:51:28Z")

</div>

Not even if we give the schema of the DataFrame when we want to write to Elasticsearch?

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [November 9, 2015, 3:57pm UTC](https://discuss.elastic.co/t/elasticsearch-2-0-and-spark-timestamptype-conversion-issue/34136/4 "2015-11-09T15:57:01Z")

</div>

No. The schema is simply Spark's representation of the data. The mapping in ES, is its own.  
The conversion in ES relies on conventions and if needed, is pluggable through `ValueReader`/`ValueWriter`.

By the way, have you tried using the `es.mapping.date.rich` [parameter](https://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#mapping) introduced in 2.1?

---

<div class="post-metadata">

**Author:** ![Lior\_Baber](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@Lior\_Baber](https://discuss.elastic.co/u/Lior_Baber)\
**Post date:** [June 29, 2016, 6:09am UTC](https://discuss.elastic.co/t/elasticsearch-2-0-and-spark-timestamptype-conversion-issue/34136/5 "2016-06-29T06:09:42Z")

</div>

it happened to me as well see my [Stackoverfollow quetsion](http://stackoverflow.com/questions/38080330/spark-write-timestamp-as-long-in-elasticsearch/38080466?noredirect=1#comment63604176_38080466)

you can see in my code that I did use the es.mapping.date.rich parameter (which suppose to be true by default)  
any new update regarding this?

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:24pm UTC](https://discuss.elastic.co/t/elasticsearch-2-0-and-spark-timestamptype-conversion-issue/34136/6 "2017-07-06T13:24:09Z")

</div>