Geoip creates everything but the geoip.location

Elastic Stack Logstash
1

Hello, I have been just banging my head on the keyboard trying to figure out why all fields are created in my event but the ".location" field

I understand that with the correct index mapping: - like so?

{

"template": [ "logstash-", "kafka-", "filebeat-" ],
"order": 0,
"version": 60002,
"index_patterns": [ "" ],
"settings": {
"index": {
"refresh_interval": "30s"
}
},
"mappings" : {
"doc" : {
"dynamic_templates" : [ {
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text", "norms" : false,
"fields" : {
"keyword" : { "type": "keyword", "ignore_above": 256 }
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date"},
"@version": { "type": "keyword"},
"source_geo" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" },
"lat" : { "type" : "half_float" },
"lon" : { "type" : "half_float" }
}
},
"destination_geo" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" },
"lat" : { "type" : "half_float" },
"lon" : { "type" : "half_float" }
}
}
}
}
}
}

...and part of my logstash config:

geoip {
  source => "[destination_ip]"
  target => "destination_geo"
  fields => ["city_name", "continent_code", "country_code2", "country_name", "region_name", "location", "latitude", "longitude"]
  #add_field => { "destination_geo.location" => [ "%{destination_geo.lat}", "%{destination_geo.lon}" ]  }
  }

Results... Still I only get events that contain all the geo fields specified to display but the ".location" , hence I can not create a tile map... ( I don't understand why there is a .location.lat and .location.lon )

image

*Additionally, if I un-comment the unneeded "add-field" line - I am met with logstash errors like this: (so I know its not right):

... argument_exception", "reason"=>"illegal latitude value [269.99999983236194] for source_geo.location"}}}}

Could anyone help?!?!?

Yes - when I do look at the specific index mapping the fields are correctly mapped on the live index...
(partial output)

          "destination_geo": {
        "dynamic": "true",
        "properties": {
          "city_name": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "continent_code": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "country_code2": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "country_name": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "ip": {
            "type": "ip"
          },
          "latitude": {
            "type": "half_float"
          },
          "location": {
            "type": "geo_point"
          },
          "longitude": {
            "type": "half_float"
          },
          "region_name": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },

To note: I am simply updating the "logstash" mapping at the dev console with a "PUT". Not useing a special mapping template name and not specifying anything in the Logstash- elasticsearch output plugin params.

Thanks!

2

I had the same issue.
Your ES template is a bit hard to read without indentation, so I'll ask:

Is the destination_geo inside or outside the main properties? It should be inside.

To me it seems to be outside. Put it together with the others, like source @timestamp, @version and source_geo

3

I don't understand why you think this is a problem. location is a geo_point, and geo_point objects contain a lat and a lon. What do you see in Kibana when you try to create a map?

4

He's missing this:

geoip

He has the destination_geo field mapping outside of properties.
edit: or doesn't. Having trouble with missing indentation.

5

i was getting an error in the red circle.. "no field with geopiont data" yada yada..

image
image.png437×749 27.3 KB

Now... just now.... it magically gives me a drop down.

but yest @atira you are correct!

more to follow, in a few... thanks guys....

6

going to fiddle with it more and update you guys, but here is a pic from y editor since the dang form didnt take my code nicely.. :slight_smile:

image
image.png795×1272 27.2 KB

7

I'm not sure but see here.

location doesn't seem to be an available choice, yet you have it in your logstash config.

8

Yes, geo field mappings are inside the main properties.

9

Well, after removing the 'fields" line from my logstash config - within the geoip params I see all the glorious events made available by the plugin. like so...
image

still I do not see the ".location" like what @atira makes an example of above.

My tile maps do find the field needed though. Magically. I just dont understand why I do not see it in both places. Discover and KibanaVisualization editor drop down for buckets.

I'll let it ride as my maps work. Thanks guys!!

10

Not sure why, but today it all displays just fine.

Could it be an index / re-index thing? shard delete/merges that needed to take place for the change to be visible?

Thanks again guys!

image

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.