# Elasticsearch Geo Distance query in PySpark DataFrame

**URL:** <https://discuss.elastic.co/t/elasticsearch-geo-distance-query-in-pyspark-dataframe/119656>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [February 13, 2018, 2:27pm UTC](https://discuss.elastic.co/t/elasticsearch-geo-distance-query-in-pyspark-dataframe/119656 "2018-02-13T14:27:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pramod\_Sripada](https://avatars.discourse-cdn.com/v4/letter/p/85e7bf/32.png) [@Pramod\_Sripada](https://discuss.elastic.co/u/Pramod_Sripada)\
**Post date:** [February 13, 2018, 2:27pm UTC](https://discuss.elastic.co/t/elasticsearch-geo-distance-query-in-pyspark-dataframe/119656/1 "2018-02-13T14:27:01Z")

</div>

I am using elasticsearch-spark connector to query data from Elasticsearch and get it into a PySpark DataFrame. Going through materials online and few Stackoverflow answers, I understood that filters will be pushed during run-time to [Elasticsearch QueryDSL](https://github.com/elastic/elasticsearch-hadoop#apache-spark). But I am not sure how to specify a [Geo Distance Query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-distance-query.html) to filter data into PySpark DataFrame.

My query currently looks like this

```
    q ="""{
      "query": {
            "bool" : {
                "must" : {
                    "match_all" : {}
                },
               "filter" : {
                    "geo_distance" : {
                        "distance" : "2.5km",
                        "location" : {
                            "lat" : 30.825,
                            "lon" : -86.99
                        }
                    }
                }
            }
        }
    }"

```

And I am currently using it to create an RDD with the following code

```
es_rdd = sc.newAPIHadoopRDD(
        inputFormatClass="org.elasticsearch.hadoop.mr.EsInputFormat",
        keyClass="org.apache.hadoop.io.NullWritable",
        valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",
        conf=es_conf)

```

\*\* I specified the query in the es\_conf dictionary

I would like to replicate the Geo Distance Query in PySpark DataFrame operation. Any help would be appreciated. Thank you.

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [March 1, 2018, 9:36pm UTC](https://discuss.elastic.co/t/elasticsearch-geo-distance-query-in-pyspark-dataframe/119656/2 "2018-03-01T21:36:40Z")

</div>

> [@Pramod\_Sripada](#):
>
> \*\* I specified the query in the es\_conf dictionary

For Geo Distance queries this is the only way to push them down at the moment. Spark does not really offer any mechanisms for translating Geo operations in their SQL dialect.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 29, 2018, 9:36pm UTC](https://discuss.elastic.co/t/elasticsearch-geo-distance-query-in-pyspark-dataframe/119656/3 "2018-03-29T21:36:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.