Error with basic geo_point setup in logstash

[jclose (Jason)]

I am trying use a basic geo_point/geoip setup in logstash.

My template for ES has the following field:
"source_geoip" : { "type" : "geo_point" }

Then, in my logstash config, I am trying the following:

            geoip {
                source => "source_ip"
                target => "source_geoip"
                add_tag => ["success_geoip"]
                tag_on_failure => ["geoip_error", "geoip_source_error"]
            }

I'm getting an error in my logstash files stating:

"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"parse_exception", "reason"=>"field must be either [lat], [lon] or [geohash]"}}}}}

I can't figure out what is going on. I can see through the rubydebug that the IPs are properly getting parsed on the logstash side. It looks like the insert into ES isn't working properly.

Can anyone give me any clue into what I need to do?

[theuntergeek (Aaron Mildenstein)]

You are sending all of the geoip data to a nested object called source_geoip. Your mapping missed the fact that location is a sub-field of that source_geoip object.

Take some cues from the default template that comes with Logstash:

        "geoip"  : {
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip" },
            "location" : { "type" : "geo_point" },
            "latitude" : { "type" : "half_float" },
            "longitude" : { "type" : "half_float" }
          }
        }

Note how geoip is an object with nested fields? The location field is the one you want to be a geo_point.

Your mapping which has

"source_geoip" : { "type" : "geo_point" }

Should look more like:

        "source_geoip"  : {
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip" },
            "location" : { "type" : "geo_point" },
            "latitude" : { "type" : "half_float" },
            "longitude" : { "type" : "half_float" }
          }
        }

[jclose (Jason)]

Thank you so much.

This should really be in some sort of official documentation, perhaps on the geoip filter page for logstash.

[theuntergeek (Aaron Mildenstein)]

Happy to help. Mapping is a rather tricky concept, though—more than a single page of documentation can cover well.

We did create a blog post to help make custom mappings with Logstash, though: https://www.elastic.co/blog/logstash_lesson_elasticsearch_mapping

If you had know about the blog post, and used the examples in the blog there as a template, you would have nailed this, I'm sure.