# Elastic spark connector not able map longitude ,latitude values to geo\_type

**URL:** <https://discuss.elastic.co/t/elastic-spark-connector-not-able-map-longitude-latitude-values-to-geo-type/60865>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [September 19, 2016, 10:07am UTC](https://discuss.elastic.co/t/elastic-spark-connector-not-able-map-longitude-latitude-values-to-geo-type/60865 "2016-09-19T10:07:47Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pavan\_Penhandra](https://avatars.discourse-cdn.com/v4/letter/p/5f8ce5/32.png) [@Pavan\_Penhandra](https://discuss.elastic.co/u/Pavan_Penhandra)\
**Post date:** [September 19, 2016, 10:07am UTC](https://discuss.elastic.co/t/elastic-spark-connector-not-able-map-longitude-latitude-values-to-geo-type/60865/1 "2016-09-19T10:07:47Z")

</div>

Hi,

When im writing longitude latitude values in string format with spark sql the data type of "location" field mapping to string not to geo\_type

I followed the format  
[https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html)

Query :sqlContext.sql( "SELECT vehicle\_id,device\_id,creation\_time,timestamp\_hour,engine\_rpm ,concat( lon,lit(","),lat) as location FROM mytable" ) .saveToEs("iot\_location/docs")

so that my string value from spark sql will be mapped to geo\_type in elastic search

but its only mapping to string.

When i tried with array of longitude ,latitude elastic search mapping it to array  
when i tried with structure (longitude ,latitude ) elastic search mapping to to structure  
I need "location" to be geo\_type so that i can use this field in kibana and grafana

location "41.12,-71.34" as string  
location -71.34, 41.12 as an array

**My index mappings**

```
    "index": {
        "mappings": {
            "sensorinfo": {
                "properties": {
                    "creation_time": {
                        "format": "strict_date_optional_time||epoch_millis",
                        "type": "date"
                    },
                    "location": {
                        "type": "geo_point",
                        "lat_lon": "true"
                    },
                    "engine_rpm": {
                        "type": "float"
                    },
                    "timestamp_hour": {
                        "format": "strict_date_optional_time||epoch_millis",
                        "type": "date"
                    },
                    "device_id": {
                        "type": "string",
                        "analyzer": "analyzer-name"
                    },
                    "vehicle_id": {
                        "type": "string",
                        "analyzer": "analyzer-name"
                    }
                }
            }
        },
       
	
			**Changed mappings after writing data to elastic**
			
			
},
"mappings": {
    "docs": {
        "properties": {
            "creation_time": {
                "format": "strict_date_optional_time||epoch_millis",
                "type": "date"
            },
            "timestamp_hour": {
                "format": "strict_date_optional_time||epoch_millis",
                "type": "date"
            },
            "device_id": {
                "type": "string"
            },
            "location": {
                "type": "string"
            },
            "engine_rpm": {
                "type": "double"
            },
            "vehicle_id": {
                "type": "string"
            }
        }
    }
},
"aliases": []

```

}

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [September 19, 2016, 6:02pm UTC](https://discuss.elastic.co/t/elastic-spark-connector-not-able-map-longitude-latitude-values-to-geo-type/60865/2 "2016-09-19T18:02:17Z")

</div>

> [@Pavan\_Penhandra](#):
>
> sqlContext.sql( "SELECT vehicle\_id,device\_id,creation\_time,timestamp\_hour,engine\_rpm ,concat( lon,lit(","),lat) as location FROM mytable" ) .saveToEs("iot\_location/docs")

Based on your post, I'm assuming that you have created a type in the index called `sensorinfo` and specified the mapping but it looks like you're writing to the `docs` type in the code example. If the `docs` type does not exist before writing, then Elasticsearch may be trying to infer the type of the location field for you (and is getting it wrong). Does the issue still happen if you write the data to the already created and mapped `iot_location/sensorinfo`?

---

<div class="post-metadata">

**Author:** ![Pavan\_Penhandra](https://avatars.discourse-cdn.com/v4/letter/p/5f8ce5/32.png) [@Pavan\_Penhandra](https://discuss.elastic.co/u/Pavan_Penhandra)\
**Post date:** [September 21, 2016, 7:24am UTC](https://discuss.elastic.co/t/elastic-spark-connector-not-able-map-longitude-latitude-values-to-geo-type/60865/3 "2016-09-21T07:24:24Z")

</div>

the issue still happening even if i write the data to the already created and mapped iot\_elastic/sensorinfo

{"iot\_elastic":  
{"mappings":  
{"sensorinfo":{"properties":{"creation\_time":{"type":"date","format":"strict\_date\_optional\_time||epoch\_millis"},"device\_id":{"type":"string"},"engine\_rpm":{"type":"double"}, **"location":{"type":"string"}** ,"timestamp\_hour":{"type":"date","format":"strict\_date\_optional\_time||epoch\_millis"},"vehicle\_id":{"type":"string"}}}}}}

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [September 21, 2016, 6:59pm UTC](https://discuss.elastic.co/t/elastic-spark-connector-not-able-map-longitude-latitude-values-to-geo-type/60865/4 "2016-09-21T18:59:47Z")

</div>

@Pavan_Penhandra, I tried to recreate this behavior on the master branch but I am unable to do so. I've tried the following code:

```
val rc = new ExtendedRestClient()

// touch the index and put the mappings
rc.put("iot_location")
rc.put("iot_location/sensorinfo/_mapping", """{"sensorinfo":{"properties":{"name":{"type": "string"},"location":{"type": "geo_point"}}}}""".getBytes(StringUtils.UTF_8))

sqlContext.createDataFrame(
    Seq(Row.fromSeq(Seq("Generic Location", "41.12,-71.34"))).asJava,
    StructType(Array(StructField("name", StringType), StructField("location", StringType)))
).saveToEs("iot_location/sensorinfo")

rc.post("iot_location/_refresh")

val df = sqlContext.read.format("es").load("iot_location")

// In SparkSQL, if location is read it is treated as a string
val dataType = df.schema("location").dataType
assertEquals("string", dataType.typeName)

// Getting the mapping should show that location is a GEO_POINT
val mappingData = RestUtils.getMapping("iot_location/sensorinfo")
assertEquals(FieldType.GEO_POINT, mappingData.properties().find(_.name() == "location").map(_.`type`()).get)

// Data exists in the index
val head = df.head()
assertThat(head.getString(0), containsString("41.12,-71.34"))
assertThat(head.getString(1), containsString("Generic Location"))

// We'll get the raw mapping response, just to double check
println("RAW MAPPING:")
println(rc.get("iot_location/_mapping"))
rc.close()

```

The output of the program is:

```
RAW MAPPING:
{"iot_location":{"mappings":{"sensorinfo":{"properties":{"location":{"type":"geo_point"},"name":{"type":"text"}}}}}}

```

Could you share which version of Elasticsearch and ES-Hadoop you are using?

---

<div class="post-metadata">

**Author:** ![Pavan\_Penhandra](https://avatars.discourse-cdn.com/v4/letter/p/5f8ce5/32.png) [@Pavan\_Penhandra](https://discuss.elastic.co/u/Pavan_Penhandra)\
**Post date:** [September 22, 2016, 1:08pm UTC](https://discuss.elastic.co/t/elastic-spark-connector-not-able-map-longitude-latitude-values-to-geo-type/60865/5 "2016-09-22T13:08:23Z")

</div>

"org.apache.spark" %% "spark-core" % "1.6.1" % "provided",  
"org.apache.spark" %% "spark-streaming" % "1.6.1" % "provided",  
"org.apache.spark" %% "spark-streaming-kafka" % "1.6.1",  
"org.elasticsearch" % "elasticsearch-spark\_2.10" % "2.3.4" % "provided",  
"org.apache.spark" % "spark-sql\_2.10" % "1.6.1",

---

<div class="post-metadata">

**Author:** ![Pavan\_Penhandra](https://avatars.discourse-cdn.com/v4/letter/p/5f8ce5/32.png) [@Pavan\_Penhandra](https://discuss.elastic.co/u/Pavan_Penhandra)\
**Post date:** [September 22, 2016, 1:09pm UTC](https://discuss.elastic.co/t/elastic-spark-connector-not-able-map-longitude-latitude-values-to-geo-type/60865/6 "2016-09-22T13:09:32Z")

</div>

{

```
"state": "open",
"settings": {
    "index": {
        "mappings": {
            "sensorinfo": {
                "properties": {
                    "creation_time": {
                        "format": "strict_date_optional_time||epoch_millis",
                        "type": "date"
                    },
                    "location": {
                        "type": "geo_point"
                    },
                    "engine_rpm": {
                        "type": "float"
                    },
                    "timestamp_hour": {
                        "format": "strict_date_optional_time||epoch_millis",
                        "type": "date"
                    },
                    "device_id": {
                        "type": "string",
                        "analyzer": "analyzer-name"
                    },
                    "vehicle_id": {
                        "type": "string",
                        "analyzer": "analyzer-name"
                    }
                }
            }
        },
        "number_of_shards": "1",
        "creation_date": "1474525784322",
        "analysis": {
            "analyzer": {
                "analyzer-name": {
                    "filter": "lowercase",
                    "type": "custom",
                    "tokenizer": "keyword"
                }
            }
        },
        "number_of_replicas": "1",
        "uuid": "Thj63TjXTMG4F8piliIP-g",
        "version": {
            "created": "2030599"
        }
    }
},
"mappings": {
    "sensorinfo": {
        "properties": {
            "creation_time": {
                "format": "strict_date_optional_time||epoch_millis",
                "type": "date"
            },
            "timestamp_hour": {
                "format": "strict_date_optional_time||epoch_millis",
                "type": "date"
            },
            "device_id": {
                "type": "string"
            },
            "location": {
                "type": "string"
            },
            "engine_rpm": {
                "type": "double"
            },
            "vehicle_id": {
                "type": "string"
            }
        }
    }
},
"aliases": []

```

}

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [September 22, 2016, 4:21pm UTC](https://discuss.elastic.co/t/elastic-spark-connector-not-able-map-longitude-latitude-values-to-geo-type/60865/7 "2016-09-22T16:21:13Z")

</div>

@Pavan_Penhandra, are you sure you are creating the index mappings correctly? It looks like you are inserting the `mappings` section into the `settings` block and it's being set as a setting instead of being recognized as a mapping. Can you create the index like so?

```
curl -XPUT localhost:9200/iot_location -d '                                                      
{
    "settings": {
        "index": {
            "number_of_shards": "1",
            "number_of_replicas": "1",
                "analysis": {
                    "analyzer": {
                        "analyzer-name": {
                        "filter": "lowercase",
                        "type": "custom",
                        "tokenizer": "keyword"
                    }
                }
            }
        }
    },
    "mappings": {
        "sensorinfo": {
            "properties": {
                "creation_time": {
                    "format": "strict_date_optional_time||epoch_millis",
                    "type": "date"
                },
                "location": {
                    "type": "geo_point"
                },
                "engine_rpm": {
                    "type": "float"
                },
                "timestamp_hour": {
                    "format": "strict_date_optional_time||epoch_millis",
                    "type": "date"
                },
                "device_id": {
                    "type": "string",
                    "analyzer": "analyzer-name"
                },
                "vehicle_id": {
                    "type": "string",
                    "analyzer": "analyzer-name"
                }
            }
        }
    }
}'
```

---

<div class="post-metadata">

**Author:** ![Pavan\_Penhandra](https://avatars.discourse-cdn.com/v4/letter/p/5f8ce5/32.png) [@Pavan\_Penhandra](https://discuss.elastic.co/u/Pavan_Penhandra)\
**Post date:** [September 23, 2016, 8:48am UTC](https://discuss.elastic.co/t/elastic-spark-connector-not-able-map-longitude-latitude-values-to-geo-type/60865/8 "2016-09-23T08:48:53Z")

</div>

@james.baiera yes you are correct .I tried with the above index and it worked .Mapping worked as expected.  
Thank you very much 🙂

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [September 23, 2016, 6:32pm UTC](https://discuss.elastic.co/t/elastic-spark-connector-not-able-map-longitude-latitude-values-to-geo-type/60865/9 "2016-09-23T18:32:30Z")

</div>

@Pavan_Penhandra Cheers!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:23pm UTC](https://discuss.elastic.co/t/elastic-spark-connector-not-able-map-longitude-latitude-values-to-geo-type/60865/10 "2017-07-06T13:23:07Z")

</div>