# PySpark - How to read timestamp date\_nanos from ElasticSearch to Spark?

**URL:** <https://discuss.elastic.co/t/pyspark-how-to-read-timestamp-date-nanos-from-elasticsearch-to-spark/260020>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [January 2, 2021, 4:04pm UTC](https://discuss.elastic.co/t/pyspark-how-to-read-timestamp-date-nanos-from-elasticsearch-to-spark/260020 "2021-01-02T16:04:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ngocdangrby](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ngocdangrby/32/81650_2.png) [@ngocdangrby](https://discuss.elastic.co/u/ngocdangrby)\
**Post date:** [January 2, 2021, 4:04pm UTC](https://discuss.elastic.co/t/pyspark-how-to-read-timestamp-date-nanos-from-elasticsearch-to-spark/260020/1 "2021-01-02T16:04:31Z")

</div>

Hi all,  
I need to read the timestamp from ElasticSearch to Spark RDD or DataFrame. I can read other fields (text, interger..) but timestamp i can not.

Spark v2.4.7  
Python: v3.6.11  
ElasticSearch: v7.10.0  
**Here is my code (python):**

```auto
    q = """{
                "query": {
                    "match_all": {}
                  }
            }"""

    es_read_conf = {
                "es.nodes": "localhost",
                "es.port": "9200",
                "es.resource": "testinterface3",
                "es.query": q,
                "es.mapping.date.rich": "false"
            }

    es_rdd = self.sc.newAPIHadoopRDD(
                inputFormatClass="org.elasticsearch.hadoop.mr.EsInputFormat",
                keyClass="org.apache.hadoop.io.NullWritable",
                valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",
                conf=es_read_conf)

     for x in es_rdd.collect():
           print(x)

```

**ES mapping:**

```auto
    "testinterface3" : {
        "aliases" : { },
        "mappings" : {
          "properties" : {
            "hello" : {
              "type" : "text"
            },
            "interface" : {
              "type" : "keyword"
            },
            "pkts" : {
              "type" : "integer"
            },
            "timestamp" : {
              "type" : "date_nanos"
            }
          }
        }

```

**Output printed:**

```auto
    ('testinterface3-0-668', {'hello': 'test', 'interface': 'testinterface3', 'pkts': 50})
    ('testinterface3-0-669', {'hello': 'test', 'interface': 'testinterface3', 'pkts': 52})
    ('testinterface3-0-670', {'hello': 'test', 'interface': 'testinterface3', 'pkts': 51})

```

**Expected output** : Has timestamp column.

Thank you in advance.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 30, 2021, 4:04pm UTC](https://discuss.elastic.co/t/pyspark-how-to-read-timestamp-date-nanos-from-elasticsearch-to-spark/260020/2 "2021-01-30T16:04:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\
**Post date:** [December 8, 2021, 4:42pm UTC](https://discuss.elastic.co/t/pyspark-how-to-read-timestamp-date-nanos-from-elasticsearch-to-spark/260020/3 "2021-12-08T16:42:17Z")

</div>

The `date_nanos` field type is unfortunately not supported yet in es-hadoop. It will be in an upcoming release -- [Unable to read date\_nanos · Issue #1653 · elastic/elasticsearch-hadoop · GitHub](https://github.com/elastic/elasticsearch-hadoop/issues/1653).