Problems with geo_point

Elastic Stack Logstash
1

Hello all,

I'm very new to elastic, so please bear with me. I'm trying to get a data type 'geo_point' in ES 6.3 via a template referenced in my logstash pipeline. I'm getting geo data populating in elasticsearch, just not a geo_point data type that's usable with the coordinate map. Here's my ES template:

{
"template" : "paloalto-",
"version" : 60001,
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"default" : {
"dynamic_templates" : [ {
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
}, {
"string_fields" : {
"match" : "",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text", "norms" : false,
"fields" : {
"keyword" : { "type": "keyword", "ignore_above": 256 }
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date"},
"@version": { "type": "keyword"},
"DestinationGeo" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
}
}
}
}
}

And here's the relevant section from my pipeline:

if [DestinationAddress] and [DestinationAddress] !~ "(^127.0.0.1)|(^10.)|(^172.1[6-9].)|(^172.2[0-9].)|(^172.3[0-1].)|(^192.168.)|(^169.254.)" {
geoip {
database => "/opt/logstash/GeoLite2-City.mmdb"
source => "DestinationAddress"
target => "DestinationGeo"
}
#Delete 0,0 in DestinationGeo.location if equal to 0,0
if ([DestinationGeo.location] and [DestinationGeo.location] =~ "0,0") {
mutate {
replace => [ "DestinationAddress.location", "" ]
}
}
}
output {
if [Type] == "TRAFFIC" {
elasticsearch {
index => "paloalto-traffic-%{DeviceName}-%{+YYYY.MM.dd}"
template => "/opt/logstash/elasticsearch-template.json"
template_overwrite => true
}
}
}

I'm not seeing any errors, so I assume I've misunderstood a piece of this. Can anyone see anything obvious missing from this configuration that would prevent the geo_point data field from populating? Do I need to install the geoip plugin on all nodes before this field will show up?

2

there are a couple of "gotcha"s here.

  • The template is just a template, used by Elasticsearch when creating new indices. Updating it does not update indices that already exist.
  • Due to the way Elasticsearch uses Lucene under the hood, it is not possible to change the type of a field in an index so you may have to delete the index and start over (or index to a new field with a new name and use it instead)
3

Hi yaauie, thanks for the reply. I've deleted the index after any changes to the template and let the index recreate applying the new template. I'm still not getting any geo_point data types in my datasets within ES. I feel like I've got something mapped wrong, but my understanding of templates is pretty limited at the moment.

4

Nothing specific is standing out to me about your template.

What is the current index mapping? The Get Mapping API is useful here.

5

It's a lot of data, too much to post. It looks like there are two fields for "DestinationGeo". I'm guessing that is my issue?

      "DestinationGeo" : {
        "dynamic" : "true",
        "properties" : {
          "ip" : {
            "type" : "ip"
          },
          "latitude" : {
            "type" : "half_float"
          },
          "location" : {
            "type" : "geo_point"
          },
          "longitude" : {
            "type" : "half_float"
          }

      "DestinationGeo" : {
        "dynamic" : "true",
        "properties" : {
          "city_name" : {
            "type" : "text",
            "norms" : false,
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "continent_code" : {
            "type" : "text",
            "norms" : false,
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "country_code2" : {
            "type" : "text",
            "norms" : false,
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "country_code3" : {
            "type" : "text",
            "norms" : false,
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "country_name" : {
            "type" : "text",
            "norms" : false,
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },

Also here's the JSON that's getting passed into ES:

"DestinationGeo": {
  "country_code2": "US",
  "region_code": "CA",
  "longitude": -122.0574,
  "latitude": 37.419200000000004,
  "city_name": "Mountain View",
  "region_name": "California",
  "country_code3": "US",
  "location": {
    "lon": -122.0574,
    "lat": 37.419200000000004
  },

But the data type is '?' for almost all of my geo data where other data types are defined. So I guess I'm missing a definer somewhere for the datatype and ES is just making a best effort?

6

Elasticsearch can't have two fields with the same name in an index, so something's not quite right; can you paste the entire mapping in a gist and link to it?

7

Sure, Here's the link. Is it because I am using the same field name in the Template that I am using in the logstash pipeline? If so, should I just change the template field to another name? I thought the template was supposed to reference the field being processed.. am I completely wrong on that?

8

The top-level wildcards indicate to me that the doc you linked to was probably your templates. Can you get the effective mapping from the specific index?

curl -X GET "${hostname}:${port}/${index}/_mapping/_doc"

We're stretching pretty firmly into Elasticsearch territory, so it may be helpful to post your questions in the Elasticsearch forum.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.