# Timestamp field being passed in epoch with Hadoop Library

**URL:** <https://discuss.elastic.co/t/timestamp-field-being-passed-in-epoch-with-hadoop-library/138774>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [July 5, 2018, 5:06pm UTC](https://discuss.elastic.co/t/timestamp-field-being-passed-in-epoch-with-hadoop-library/138774 "2018-07-05T17:06:32Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wayne\_Taylor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wayne_taylor/32/45984_2.png) [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\
**Post date:** [July 5, 2018, 5:06pm UTC](https://discuss.elastic.co/t/timestamp-field-being-passed-in-epoch-with-hadoop-library/138774/1 "2018-07-05T17:06:33Z")

</div>

HI Team,

I have been able to follow instructions to get my ORC data passed to Elasticsearch but having issues with my data source not having a timestamp and even with formatting the timestamp shows as numeric.

Below are my steps:

1. Load pyspark and pass in the Elasticsearch Hadoop JAR:  
`Downloads/spark/bin/pyspark --jars ~/Downloads/elasticsearch-hadoop-6.3.0.jar`
2. Create a data frame from a local ORC file: df = `spark.read.format("orc").load("/Users/wtaylor/Downloads/TEST/*")`
3. Create a Temp Table so I can query my ORC and aggregate:  
`usage = df.registerTempTable("esexample")`
4. Cache results from temp from my SQL: aggUrldf = spark.sql(aggSql).cache()

Note in the SQL my date source field is in Epoch with MS but I change to timestamp:  
`timestamp(from_unixtime(start_time/1000)) as start_time`

1. I then pass to ES using following:  
`aggUrldf.write.format("org.elasticsearch.spark.sql").option("es.nodes.wan.only","true").option("es.nodes", esUrl).mode("Overwrite").option("es.net.http.auth.user",esUser).option("es.net.http.auth.pass",esPassword).save("indexname/doctype")`

Verified my data is in ES. But format is numeric in Epoch. See example:

> {  
> "\_index": "indexname",  
> "\_type": "test",  
> "\_id": "qFdja2QBCIhbyqjdz7hd",  
> "\_score": 1,  
> "\_source": {  
> "id": "21590385",  
> "origination\_airport": "KCLT",  
> "destination\_airport": "KSEA",  
> "start\_time": 1530444648000,  
> "client\_ip": "10.34.11.162",  
> "url": "[gateway.icloud.com](http://gateway.icloud.com)",  
> "rx\_total\_bytes": 828,  
> "tx\_total\_bytes": 2578  
> }

I was unable to get a combination from [Configuration | Elasticsearch for Apache Hadoop [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#cfg-multi-writes-format) working.

Any ideas?

Thanks  
Wayne

---

<div class="post-metadata">

**Author:** ![Wayne\_Taylor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wayne_taylor/32/45984_2.png) [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\
**Post date:** [July 17, 2018, 12:33pm UTC](https://discuss.elastic.co/t/timestamp-field-being-passed-in-epoch-with-hadoop-library/138774/2 "2018-07-17T12:33:18Z")

</div>

After working with ES team in git this is a bug. [https://github.com/elastic/elasticsearch-hadoop/issues/1173](https://github.com/elastic/elasticsearch-hadoop/issues/1173)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 14, 2018, 12:33pm UTC](https://discuss.elastic.co/t/timestamp-field-being-passed-in-epoch-with-hadoop-library/138774/3 "2018-08-14T12:33:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.