# Create geo\_point from nested coordinates

**URL:** <https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835>\
**Category:** Logstash\
**Created:** [September 19, 2016, 6:43am UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835 "2016-09-19T06:43:05Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Crai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crai/32/11629_2.png) [@Crai](https://discuss.elastic.co/u/Crai)\
**Post date:** [September 19, 2016, 6:43am UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835/1 "2016-09-19T06:43:05Z")

</div>

I am trying to create a geo\_point field to add coords using the http poller to pull some data in json format.

so the json is coming in a format like this. Multiple JSON documents representing each item and it's properties and sub-properties

```
[
    {
        "1": "a",
        "2": "",
        "3": "",
        "4": {
        "4a": {
            "4a1": ""
        },
        "4b": {
            "4b1": ""
        },
        "4c": {
            "latitude": 01,
            "longitude": -01
        }
      }
    },
    {
        "1": "b",
        "2": "",
        "3": "",
        "4": {
        "4a": {
            "4a1": ""
        },
        "4b": {
            "4b1": ""
        },
        "4c": {
            "latitude": 01,
            "longitude": -01
        }
      }
    }
]

```

so for my config i have

```
input {
  http_poller {
    urls => {
      url => {
        method => get
        url => ""
        headers => {
          Accept => "application/json"
        }
      }
    }
    request_timeout => 60
    interval => 60
    codec => "json"
    metadata_target => "http_poller_metadata"
  }
}

filter {
    mutate {
      add_field => { "[location][lat]" => "%{[4][4c][latitude]}" }
      add_field => { "[location][lon]" => "%{[4][4c][longitude]}" }
    }
    mutate {
      convert => { "[location]" => "geo_point" }
    }
    mutate {
      convert => { "[location][lat]" => "float" }
      convert => { "[location][lon]" => "float" }
    }
  }

output {
  elasticsearch { 
    hosts => ["localhost:9200"]
    manage_template => false
  }
  stdout { codec => rubydebug }
}

```

The http poller pulls the data in wonderfully without the mutate. i can also mutate and add a field such as test\_field: test\_data without issue. anything I start using anything regarding nested data it falls on its face. So, I'm trying to figure out where my formatting is off for the filters. is [4] the true top-level field or is it the item in the array? or do i need to split them first?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 19, 2016, 7:53pm UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835/2 "2016-09-19T19:53:00Z")

</div>

Since you have an array of objects with lat/lon pairs I'd assume that you'll want to split the array with the split filter.

> ```
> convert => { "[location]" => "geo_point" }
> 
> ```

`geo_point` isn't a valid type to convert to.

---

<div class="post-metadata">

**Author:** ![Crai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crai/32/11629_2.png) [@Crai](https://discuss.elastic.co/u/Crai)\
**Post date:** [September 20, 2016, 6:28am UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835/3 "2016-09-20T06:28:05Z")

</div>

ok thanks for the info. so I would need to change the mapping for the index to get the type changed to geo\_type. ok... so for the split. this is where I was also unsure. I thought it was the route I needed to go. It seems as though the split works fine, but adding nested fields is where i seem to be failing.

```
filter {
  split {
    field => "id"
  }
  mutate {
    add_field => { "location" => "test_loc" }
    add_field => { "[location][lat]" => "test_lat" }
    add_field => { "[location][lon]" => "test_lon" }
  }
}

```

I tried having two mutates one wit hadding location and the other location lat and lon... adding just location works, but adding the lat and lon sub fields fails.

sample from the massive error...

> "Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.", "exception"=\>java.lang.ClassCastException: expecting List or Map, found class com.logstash.bivalues.StringBiValue, "backtrace"=\>["com.logstash.Accessors.newCollectionException(Accessors.java:175)", "com.logstash.Accessors.store(Accessors.java:162)",

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 20, 2016, 6:31am UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835/4 "2016-09-20T06:31:19Z")

</div>

> ```
> add_field => { "location" => "test_loc" }
> 
> ```

Here you add `location` as a string value.

> ```
> add_field => { "[location][lat]" => "test_lat" }
> 
> ```

Here you want to add `lat` as a subfield of `location`, but you just created `location` as a string value.

---

<div class="post-metadata">

**Author:** ![Crai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crai/32/11629_2.png) [@Crai](https://discuss.elastic.co/u/Crai)\
**Post date:** [September 20, 2016, 7:05am UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835/5 "2016-09-20T07:05:16Z")

</div>

ok... so i got a bit further it looks like. added the fields successfully and the output from ES.

```
filter {
  split {
    field => "id"
  }
  mutate {
    add_field => { "[location][lat]" => 0 }
    add_field => { "[location][lon]" => 1 }
  }
}

```

* * *

```
"location": {
      "lon": 1,
      "lat": 0
    },

```

but this doesn't...

```
  mutate {
        add_field => { "[vehicle][location][lat]" => 0 }
        add_field => { "[vehicle][location][lon]" => 1 }
      }

```

the structure is correct from what i can see... so I am stumped. essentially, I'm wanting to take the original longitude and latitude values and put them in a geo\_point field so i can map them in kibana.

```
{
    "id": "8416",
    "is_deleted": false,
    "trip_update": null,
    "vehicle": {
      "trip": {
        "trip_id": "UP-N_UN368_V1_C",
        "route_id": "UP-N",
        "direction_id": null,
        "start_time": "23:35:00",
        "start_date": "20160920",
        "schedule_relationship": 0
      },
      "vehicle": {
        "id": "8416",
        "label": "368",
        "license_plate": null
      },
      "position": {
        "latitude": 41.93284225463867,
        "longitude": -87.67341613769531,
        "bearing": null,
        "odometer": null,
        "speed": null
      },
      "current_stop_sequence": null,
      "stop_id": null,
      "current_status": 2,
      "timestamp": {
        "low": "2016-09-20T06:03:13.000Z",
        "high": 0,
        "unsigned": true
      },
      "congestion_level": null,
      "occupancy_status": null
    },
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 20, 2016, 8:32am UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835/6 "2016-09-20T08:32:34Z")

</div>

Which field in ES do you want to have as a geo\_point field? How is that field currently mapped?

---

<div class="post-metadata">

**Author:** ![Crai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crai/32/11629_2.png) [@Crai](https://discuss.elastic.co/u/Crai)\
**Post date:** [September 20, 2016, 10:33pm UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835/7 "2016-09-20T22:33:04Z")

</div>

the location field is what i want to map, but i haven't yet because i was having an issue with all of the mutation stuff. so right now it's not mapped at all.

---

<div class="post-metadata">

**Author:** ![Crai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crai/32/11629_2.png) [@Crai](https://discuss.elastic.co/u/Crai)\
**Post date:** [September 21, 2016, 5:30am UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835/8 "2016-09-21T05:30:57Z")

</div>

this is my first time trying to do my own mapping in an index and i cannot seem to get the mapping to take. This is what i'm trying to do with no success.

```
PUT _template/logstash/
{
  "mappings": {
      "properties": {
          "vehicle:" {
          "location": {
            "type": "geo_point"
          }
        }
      }
    }
  }
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 21, 2016, 5:37am UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835/9 "2016-09-21T05:37:36Z")

</div>

You need one `properties` key for each level.

```nohighlight
{
  "mappings": {
    "properties": {
      "vehicle:" {
        "properties": {
          "location": {
            "type": "geo_point"
          }
        }
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![Crai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crai/32/11629_2.png) [@Crai](https://discuss.elastic.co/u/Crai)\
**Post date:** [September 21, 2016, 10:11pm UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835/10 "2016-09-21T22:11:16Z")

</div>

So... I think I have it...

```
PUT _template/logstash/
{
  "template": "logstash-*",
  "mappings": {
     "_default_" : { 
       "properties" : { 
         "vehicle": {
           "properties": {
         "location": { "type": "geo_point" }
           }
        }
      }
    }
  }
}

```

* * *

```
{
  "acknowledged": true
}

```

* * *

```
GET _template/logstash/
	
	{
  "logstash": {
    "order": 0,
    "template": "logstash-*",
    "settings": {},
    "mappings": {
      "_default_": {
        "properties": {
          "vehicle": {
            "properties": {
              "location": {
                "type": "geo_point"
              }
            }
          }
        }
      }
    },
    "aliases": {}
  }
}
```

---

<div class="post-metadata">

**Author:** ![Crai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crai/32/11629_2.png) [@Crai](https://discuss.elastic.co/u/Crai)\
**Post date:** [September 21, 2016, 10:28pm UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835/11 "2016-09-21T22:28:47Z")

</div>

seems to be failing on the filter still

```
filter {
  split {
    field => "id"
  }
  mutate {
    add_field => { "[vehicle][location][lat]" => "%{vehicle.position.latitude}" }
    add_field => { "[vehicle][location][lon]" => "%{vehicle.position.longitude}" }
  }
  mutate {
    convert => {"[vehicle][location][lat]" => "float"}
    convert => {"[vehicle][location][lon]" => "float"}
  }
}

```

> "2016-09-21T17:17:33.515000-0500", :message=\>"Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.", "exception"=\>java.lang.ClassCastException: expecting List or Map, found class com.logstash.bivalues.NullBiValue, "backtrace"=\>["com.logstash.Accessors.newCollectionException(Accessors.java:175)", "com.logstash.Accessors.fetch(Accessors.java:139)", "com.logstash.Accessors.findCreateTarget(Accessors.java:93)", "com.logstash.Accessors.set(Accessors.java:25)", "com.logstash.Event.setField(Event.java:156)", "com.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby\_set\_field(JrubyEventExtLibrary.java:144)",

---

<div class="post-metadata">

**Author:** ![Crai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crai/32/11629_2.png) [@Crai](https://discuss.elastic.co/u/Crai)\
**Post date:** [September 21, 2016, 10:54pm UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835/12 "2016-09-21T22:54:48Z")

</div>

in order to simplify i moved things a bit...

```
filter {
  split {
    field => "id"
  }
   mutate {
    add_field => { "location" => "%{[vehicle][position][latitude]}" }
    add_field => { "location" => "%{[vehicle][position][longitude]}" }
  }
}

{
  "logstash": {
    "order": 0,
    "template": "logstash-*",
    "settings": {},
    "mappings": {
      "_default_": {
        "properties": {
          "location": {
            "type": "geo_point"
          }
        }
      }
    },
    "aliases": {}
  }
}

```

here is my error

> {:timestamp=\>"2016-09-21T17:51:44.503000-0500", :message=\>"Failed action.", :status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"logstash-2016.09.21", :\_type=\>"logs", :\_routing=\>nil}, 2016-09-21T22:51:43.541Z %{host} %{message}], :response=\>{"index"=\>{"\_index"=\>"logstash-2016.09.21", "\_type"=\>"logs", "\_id"=\>"AVdO8plc2ExhXBmCnMs9", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse", "caused\_by"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"illegal latitude value [269.99999983236194] for location"}}}}, :level=\>:warn}

---

<div class="post-metadata">

**Author:** ![Crai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crai/32/11629_2.png) [@Crai](https://discuss.elastic.co/u/Crai)\
**Post date:** [September 21, 2016, 11:16pm UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835/13 "2016-09-21T23:16:39Z")

</div>

finally got it! man what a learning experience. haha

```
{
  "logstash": {
    "order": 0,
    "template": "logstash-*",
    "settings": {},
    "mappings": {
      "_default_": {
        "properties": {
          "location": {
            "type": "geo_point"
          }
        }
      }
    },
    "aliases": {}
  }
}

filter {
  split {
    field => "id"
  }
   mutate {
   add_field => { "[location][lat]" => "%{[vehicle][position][latitude]}" }
   add_field => { "[location][lon]" => "%{[vehicle][position][longitude]}" }
  }
   mutate {
    convert => {"[location][lat]" => "float"}
    convert => {"[location][lon]" => "float"}
   }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:37am UTC](https://discuss.elastic.co/t/create-geo-point-from-nested-coordinates/60835/14 "2017-07-06T04:37:29Z")

</div>