Problems with mapping data as geo_point datatype

[olliewardman (Oliver Wardman)]

Hi.

elasticsearch 5.1.1
kibana 5.1.1
logstash 5.1.1

I am capturing tweets using the logstash twitter input plugin and passing them to elasticsearch using the elasticsearch output plugin. All works well except I seem unable to correctly map coordinates as the geo_point datatype.

I append my mapping template below. This is based on David Pilato's excellent article Indexing Twitter with Logstash and Elasticsearch (http://david.pilato.fr/blog/2015/06/01/indexing-twitter-with-logstash-and-elasticsearch/).

I know I have coordinates in my elasticsearch index but they are using the float datatype not geo_point despite the mapping below.

Any help would be much appreciated.

Kind regards.

Oliver

{
  "template": "twitter",
  "order": 1,
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "tweet": {
      "_all": {
        "enabled": false
      },
      "dynamic_templates": [
        {
          "message_field": {
            "match": "message",
            "match_mapping_type": "string",
            "mapping": {
              "type": "string",
              "index": "analyzed",
              "omit_norms": true
            }
          }
        },
        {
          "string_fields": {
            "match": "*",
            "match_mapping_type": "string",
            "mapping": {
              "type": "string",
              "index": "analyzed",
              "omit_norms": true,
              "fields": {
                "raw": {
                  "type": "string",
                  "index": "not_analyzed",
                  "ignore_above": 256
                }
              }
            }
          }
        }
      ],
      "properties": {
        "text": {
          "type": "string"
        },
        "coordinates": {
          "properties": {
            "coordinates": {
              "type": "geo_point"
            },
            "type": {
              "type": "string"
            }
          }
        }
      }
    }
  }
}

[dadoonet (David Pilato)]

Thank you for the comment about the blog

Can you show as well an example of a Tweet document?

[dadoonet (David Pilato)]

Here is the template I'm using with 5.0 BTW:

{
  "template": "twitter*",
  "order":    1,
  "settings": {
    "number_of_shards": 1,
    "mapping.total_fields.limit": 1800
  },
  "mappings": {
    "tweet": {
      "properties": {
        "text": {
          "type": "text"
        },
        "coordinates": {
          "properties": {
             "coordinates": {
                "type": "geo_point"
             },
             "type": {
                "type": "text"
             }
          }
       }
      }
    }
  }
}

[olliewardman (Oliver Wardman)]

Thanks David.

I append a tweet document below using your new template. This tweet contains coordinates (its one of mine). Does the document look as you would expect? I note that the Tile Map visualisation still doesn't find any geo_point types in the data.

Kind regards.

Oliver

{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "tweet2-2016.12.28",
"_type": "tweet",
"_id": "AVlHHdi-goIu1XeNUTpB",
"_score": 1,
"_source": {
"in_reply_to_status_id_str": null,
"in_reply_to_status_id": null,
"created_at": "Wed Dec 28 20:27:26 +0000 2016",
"in_reply_to_user_id_str": null,
"source": "<a href="http://tapbots.com/tweetbot" rel="nofollow">Tweetbot for iΟS",
"retweet_count": 0,
"retweeted": false,
"geo": {
"coordinates": [
53.97239888,
-1.04706316
],
"type": "Point"
},
"filter_level": "low",
"in_reply_to_screen_name": null,
"is_quote_status": false,
"id_str": "814206142326992896",
"in_reply_to_user_id": null,
"@version": "1",
"favorite_count": 0,
"id": 814206142326992900,
"text": "Testing",
"place": {
"country_code": "GB",
"country": "United Kingdom",
"full_name": "York, England",
"bounding_box": {
"coordinates": [
[
[
-1.147268,
53.928002
],
[
-1.147268,
54.004411
],
[
-1.023381,
54.004411
],
[
-1.023381,
53.928002
]
]
],
"type": "Polygon"
},
"place_type": "city",
"name": "York",
"attributes": {},
"id": "0c4f58af36a910a6",
"url": "https://api.twitter.com/1.1/geo/id/0c4f58af36a910a6.json"
},
"lang": "en",
"favorited": false,
"coordinates": {
"coordinates": [
-1.04706316,
53.97239888
],
"type": "Point"
},
"truncated": false,
"timestamp_ms": "1482956846206",
"tags": [],
"@timestamp": "2016-12-28T20:27:26.000Z",
"entities": {
"urls": [],
"hashtags": [],
"user_mentions": [],
"symbols": []
},
"contributors": null,
"user": {
"utc_offset": 0,
"friends_count": 77,
"profile_image_url_https": "https://pbs.twimg.com/profile_images/2637684812/8d6746d79f6aef88c4af48511162d90c_normal.jpeg",
"listed_count": 8,
"profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png",
"default_profile_image": false,
"favourites_count": 9,
"description": "Occasional musings on tech. Views my own.",
"created_at": "Wed Jun 15 18:57:55 +0000 2011",
"is_translator": false,
"profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png",
"protected": false,
"screen_name": "olliewardman",
"id_str": "317958440",
"profile_link_color": "1DA1F2",
"id": 317958440,
"geo_enabled": true,
"profile_background_color": "C0DEED",
"lang": "en",
"profile_sidebar_border_color": "C0DEED",
"profile_text_color": "333333",
"verified": false,
"profile_image_url": "http://pbs.twimg.com/profile_images/2637684812/8d6746d79f6aef88c4af48511162d90c_normal.jpeg",
"time_zone": "London",
"url": null,
"contributors_enabled": false,
"profile_background_tile": false,
"statuses_count": 245,
"follow_request_sent": null,
"followers_count": 19,
"profile_use_background_image": true,
"default_profile": true,
"following": null,
"name": "Oliver Wardman",
"location": "York, U.K.",
"profile_sidebar_fill_color": "DDEEF6",
"notifications": null
}
}
}
]
}
}