Missing geoip.location

[brapnda23 (Brajesh Panda)]

New to ELK...My test ELK instance is 7.8.x.

I am using the below filter code for IIS Logs and it is working fine. I can see other geo fields but I don't see geoip.location as mentioned in https://www.elastic.co/blog/geoip-in-the-elastic-stack. Anything else I should be doing to get the geoip.location?

filter {
  if "IIS" in [tags] {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{NOTSPACE:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} %{NOTSPACE:CS-URI-Query} %{NUMBER:S-Port} %{NOTSPACE:CS-Username} %{IPORHOST:C-IP} %{NOTSPACE:CS-Version} %{NOTSPACE:CS-UserAgent} %{NOTSPACE:CS-Cookie} %{NOTSPACE:CS-Referer} %{NOTSPACE:CS-Host} %{NUMBER:SC-Status} %{NUMBER:SC-SubStatus} %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}"}
    }
geoip { source => "C-IP"}
  }
}

[Jenni]

I think I don't understand your question? Where are you missing the location? In your screenshot it is listed: geoip.location.lat and geoip.location.lon. Do you mean that is not listed as a geopoint data type? That would probably be because you didn't define it as a geo_point in your mapping.

[brapnda23 (Brajesh Panda)]

That screenshot is from kibana discover. I thought geoip.location should be another filed over there along with .lat and .lon. Like the below screenshot from the above article.

[Jenni]

geoip.location is a field that consists of two values: lat and lon. To use it in the map visualization it has to be defined as the data type geo_point in the mapping of your index (the definition of the data types of all the fields in your index). This is shown in your tutorial in the section "Mapping, for Maps".

"geoip"  : {
  "dynamic": true,
  "properties" : {
    ...
    "location" : { "type" : "geo_point" },
    ...
  }

If you are not using one of those default templates mentioned, you'll have to configure this data type yourself:
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html

You can see the current mapping by querying GET yourindex/_mapping. To index your data with a different data type you'll have to reindex it in a new index with the correct settings because you can not change the mapping of a field once it has already been assigned a data type.

[andrew.campbell]

Hey Jenni, same issue for me. But I am not seeing how to deal with the mapping as I have the data coming from Logstash and when creating the Index Pattern, there is no way to change field mappings before the indexing happens. Or are you saying this is done in Logstash? I have indicies for each day's worth of logs and I see that these fields are mapped to a number instead of the geo_point. I can change this in the Index Pattern after the fact and upload it (via Saved Objects), but that won't help with the current index as you mention. But, if I create a new index from Logstash, that new Index Pattern will be used and back at square one where the mapping is incorrect.

As a side comment, it seems odd that these values don't map to geo_point out of the box as that is what they are used for...

Anyway, thank you!

[Jenni]

@andrew.campbell

there is no way to change field mappings before the indexing happens

That's exactly what index templates are for

Templates are configured prior to index creation and then when an index is created either manually or through indexing a document, the template settings are used as a basis for creating the index

[andrew.campbell]

OK, I've been able to setup the index template and the fields are now set to a type of geo_point. But now i'm getting a parse error. geo_point expected, but that is what is the configured type.

:response=>{"index"=>{"_index"=>"colliers.com-prod-cd-2020.06.11", "_type"=>"_doc", "_id"=>"4Hp7VHMB5Vmbl9eI_mv9", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [geoip.location.lon] of type [geo_point]", "caused_by"=>{"type"=>"parse_exception", "reason"=>"geo_point expected"}

This is the specific section of the template:
"latitude" : { "type" : "float" }, "location" : { "properties" : { "lat" : { "type" : "geo_point" }, "lon" : { "type" : "geo_point" } } }, "longitude" : { "type" : "float" },

I originally had the full latitude and longitude fields set to geo_point as well, and that also failed parsing.

What am I missing? thanks for your help with this!

[Badger]

@andrew.campbell

"failed to parse field [geoip.location.lon] of type [geo_point]"

If your template sets geoip.location.lon as a geo_point then remove that. geoip.location should be a geo_point.

[andrew.campbell]

Thanks for the help all! Issue resolved...in a different way. I was unable to get the template working from the dev tools. I could request it and see that it was updated, but it wasn't being used to parse the logs. I ended up creating it via the gui to create an new index template and add the same json for the mappings. That worked and now I have the location type of geo_point on the map

Still not sure why updating via the dev tools did not work and through the gui did. But, removing the .lat and .lon fields and setting only location did do the trick.

        "latitude": {
          "type": "float"
        },
        "location": {
          "type": "geo_point"
        },
        "longitude": {
          "type": "float"
        },

@brapnda23 I can show you the details tomorrow...