Unable to view geo.location on Kibana even though I have created an index template that includes geo.location

Elastic Stack Kibana
1

I am using Logstash's geoip filter to get location information from ip.

As far as I can see from the Logstash output result (codec => rubydebug), I am getting geoip.location, but I cannot display it on Kibana.

In order to display it in Kibana, I have created an index template and set geoip.location to type geo_point.

{
... snip ...
 "geo_index_template" : {
    "order" : 0,
    "index_patterns" : [
      "*-www-access-*",
      "*-nginx-access-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "1"
        "number_of_replicas" : "0"
      }
    },
    "mappings" : {
      "_source" : {
        "enabled" : true
      },
      "properties" : {
        "geoip.location" : {
          "type" : "geo_point"
        }
      }
    },
    "aliases" : { }
  },
... snip ...
}

The index I am creating matches the index pattern in the index template above, but I don't see geo.location on Kibana.
Why is this?

Any advice would be greatly appreciated.

image

The mapping of the indexes I am getting is as follows


{
  "kin-www-access-2022.08.05" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
      
      ... snip ...
      
        "geoip" : {
          "properties" : {
          
          ... snip ...
          
            "location" : {
              "properties" : {
                "lat" : {
                  "type" : "float"
                },
                "lon" : {
                  "type" : "float"
                }
              }
            },
          
          ... snip ...
          
}
2

Hello,

could you share more details, for example the Logstash's pipeline and an event?

3

Thanks for your reply.
I have the following statement in my logstash configuration file.

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => {
      "message" => '%{IP:nginx.access.remote_ip_list}\s%{DATA:nginx.access.user_name}\s%{DATA:nginx.access.user}\s\[%{HTTPDATE:nginx.access.timestamp}\]\s\"%{WORD:nginx.access.method}\s%{DATA:nginx.access.url}\sHTTP/%{DATA:nginx.access.http_version}\"\s%{NUMBER:nginx.access.response_code}\s%{NUMBER:nginx.access.body_sent.bytes:int}\s\"%{DATA:nginx.access.referrer}\"\s\"%{DATA:nginx.access.agent}"'
    }
  }
  useragent {
      source => "nginx.access.agent"
      target => "nginx.access.useragent"
  }
  geoip {
      source => "nginx.access.remote_ip_list"
      #target => "nginx.access.geoip"
  }
}
output {
  elasticsearch {
    hosts => ["localhost"]
    index => "%{[fields][index_name]}-%{+YYYY.MM.dd}"
  }
  #file {
  #   path => "/var/log/logstash/my_output_text_file.txt"
  #   codec => rubydebug
  #}
}

The result of codec => rubydebug looks like this
I think I got the value well here.

{
... snip ...

      "nginx.access.response_code" => "200",
                           "geoip" => {
          "country_name" => "United States",
         "country_code2" => "US",
             "longitude" => -XX.XXX,
                    "ip" => "XXX.XXX.XXX.XXX",
         "country_code3" => "US",
              "timezone" => "America/Chicago",
        "continent_code" => "NA",
              "location" => {
            "lat" => XX.XXX,
            "lon" => XX.XXX
        },
              "latitude" => XX.XXX
    },

... snip ...

}
4

I think also. Geoip plugin is working.

I would suggest two options:

  1. What version of kibana & elastic are you using? I think before 8.0.0 the index pattern could not be refreshed and probably does not contain all your fields mapped. If that, you could update under management in Kibana. Update index pattern API | Kibana Guide [8.11] | Elastic
  2. The mapping is not correct and it's not mapping correctly your field geoip.location
5

I am using version 7.12.
The page you gave me does not seem to exist in 7.12.

Should I use the following format?

  "mappings": {
    "properties": {
      "geoip": {
        "properties": {
          "location": {
            "type": "geo_point"
          }
        }
      }
    }
  }
6

Probably you'll have to refresh your index pattern.

To check if, go to Index Patterns (under management in Kibana) and check if the geoip.location field is recognised as geo_point. You should see something like this:

geo
geo906×80 3.88 KB

7

image
image1287×108 5.08 KB

No, geoip.location is not recognized.
Instead, geoip.location.lat and geoip.location.lon are available.

How can I make geoip.location be recognized?

8

Yes, so it's not recognized.

By correcting the mapping

to this.

If you have data indexed I think you cannot change the mapping of existing fields, but you can add a new one Update mapping API | Elasticsearch Guide [8.11] | Elastic. If you can, remove the index, mapping, index pattern and recreate, ensuring it takes the new values.

9

I have re-created the mapping based on the page you gave me.
Is the following correct?
However, it fails.

# curl -X PUT "localhost:9200/kin-www-access-2022.08.05/_mapping?pretty" -H 'Content-Type: application/json' -d'
{
  "properties": {
    "geoip": {
      "properties": {
        "location": {
          "type": "geo_point"
        }
      }
    }
  }
}
'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "can't merge a non object mapping [geoip.location] with an object mapping"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "can't merge a non object mapping [geoip.location] with an object mapping"
  },
  "status" : 400
}

As it should be, the geoip.location is not created in the index.

# curl -X GET "localhost:9200/kin-www-access-2022.08.05/_mapping?pretty"
{
  "kin-www-access-2022.08.05" : {
    "mappings" : {
      "properties" : {
      ... snip ...
            "location" : {
              "properties" : {
                "lat" : {
                  "type" : "float"
                },
                "lon" : {
                  "type" : "float"
                }
              }
            },
      ... snip ...

How can I make two float types geo_point?

Note that it is difficult to delete the index.

The date changed and a new index was created, but geoip.location was not created.
Is there something wrong with my settings?

10

There are a couple issues in your case.

First, the geoip filter in logstash will, by default, store the location data in the location property of the geoip filter, it will also create two additional fields under location, lat and lon, so in the end you will have: geoip.location.lat and geoip.location.lon.

The mapping for this field needs to be:

{
  "mappings": {
    "properties": {
      "geoip": {
        "properties": {
          "location": {
            "type": "geo_point"
          }
        }
      }
    }
  }

Your first mapping was:

      "properties" : {
        "geoip.location" : {
          "type" : "geo_point"
        }
      }

Which is not the same, in this case you have a field named geoip.location where the dot is part of the name.

The error you got when you tried to apply the new mapping seems to be a conflict with your old mapping, mixing fields with dot in the name with similar named json objects can give you some issues, in fact, you should avoid using dot in fields names.

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "can't merge a non object mapping [geoip.location] with an object mapping"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "can't merge a non object mapping [geoip.location] with an object mapping"
  },
  "status" : 400
}

Try the following steps to fix your issue:

  1. Stop the indexing
  2. Delete the current index
  3. Apply your correct mapping or template
  4. Start the indexing again

This should make your geoip work. If you want a practical example, I have an old blog post on how to use geoip in Logstash.

Another issue that you have, but is not related with geoip, is that your filters are creating fields with dot in the name and this can be confusing since the notation is different in Logstash and Elasticsearch/Kibana.

In Elasticsearch/Kibana a nested field like this one: field.nestedfield means that you have this structure:

{ "field": { "nestedfield": "value" } }

But in Logstash, if you use field.nestedfield it will give you this structure:

{ "field.nestedfield": "value" }

Those things are different, but in Kibana they will look the same in some places and this can lead to confusion.

In Logstash the correct way to create and work with nested fields is using brackets.

So, the fields in your grok and other filters should be like this:

[nginx][access][remote_ip_list]
[nginx][access][user_name]
[nginx][access][user]
[nginx][access][timestamp]
1 Like
11

I have rewritten the template and filter descriptions.
Is this what you intend?

# vi /etc/logstash/logstash-its.conf

... snip ... 

filter {
    grok {
        match => {
            "message" => '%{IP:[nginx][access][remote_ip_list]}\s%{DATA:[nginx][access][user_name]}\s%{DATA:[nginx][access][user]}\s\[%{HTTPDATE:[nginx][access][timestamp]}\]\s\"%{WORD:[nginx][access][method]}\s%{DATA:[nginx][access][url]}\sHTTP/%{DATA:[nginx][access][http_version]}\"\s%{NUMBER:[nginx][access][response_code]}\s%{NUMBER:[nginx][access][body_sent][bytes]:int}\s\"%{DATA:[nginx][access][referrer]}\"\s\"%{DATA:[nginx][access][agent]}"'
        }
    }
    useragent {
        source => "[nginx][access][agent]"
        target => "[nginx][access][useragent]"
    }
    geoip {
        source => "[nginx][access][remote_ip_list]"
    }
}

... snip ... 


# curl -X GET "localhost:9200/_index_template/geo_index_template?pretty"
{
  "index_templates" : [
    {
      "name" : "geo_index_template",
      "index_template" : {
        "index_patterns" : [
          "*-www-access-*",
          "*-nginx-access-*"
        ],
        "template" : {
          "settings" : {
            "index" : {
              "number_of_shards" : "1",
              "number_of_replicas" : "0"
            }
          },
          "mappings" : {
            "properties" : {
              "geoip" : {
                "properties" : {
                  "location" : {
                    "type" : "geo_point"
                  }
                }
              }
            }
          }
        },
        "composed_of" : [ ]
      }
    }
  ]
}

However, it is not currently working well.
This is because we are not re-creating the index.

Stopping or deleting indexes is not casually possible at present.
There are people who refer to these.

I create each index daily in the format ****-%{+YYYY.MM.dd}.
Would this not allow me to get the location from that morning as the new index is created the next day?

Soon a new index with a new date will be created.
Hope that works, but I am a little skeptical. Because I was not able to get location well in this morning's index.
I will let you know the result of this tomorrow.

12

Since you have a template for your indices that is creating the correct mapping for the geoip field, then it will work for the new indices.

13

I tried waiting for a new index to be created.

However, location is not recognized as geo_point.

Is there anything else I can think of?

# curl -X GET localhost:9200/kin-www-access-2022.08.07?pretty
{
  "kin-www-access-2022.08.07" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
... snip ...
        "geoip" : {
          "properties" : {
... snip ...
            "location" : {
              "properties" : {
                "lat" : {
                  "type" : "float"
                },
                "lon" : {
                  "type" : "float"
                }
              }
            },
... snip ...
14

I have tried to create the index template several times over the past few days, but all have failed.
Yesterday, I re-created the index template and it worked and the location is now recognized as a geo_point in today's log.
Here is the index template I created yesterday.

curl -X PUT "localhost:9200/_index_template/access_index_template?pretty" -H 'Content-Type: application/json' -d'
{
  "index_patterns" : ["*-www-access-*","*-httpd-access-*"],
  "priority" : 2,
  "template": {
    "settings" : {
      "number_of_shards": 1,
      "number_of_replicas": 0
    },
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "agent" : {
          "properties" : {
            "ephemeral_id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "ecs" : {
          "properties" : {
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "fields" : {
          "properties" : {
            "index_name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "geoip" : {
          "properties" : {
            "city_name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "continent_code" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "country_code2" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "country_code3" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "country_name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "dma_code" : {
              "type" : "long"
            },
            "ip" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "latitude" : {
              "type" : "float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude" : {
              "type" : "float"
            },
            "postal_code" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "region_code" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "region_name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "timezone" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "host" : {
          "properties" : {
            "architecture" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "containerized" : {
              "type" : "boolean"
            },
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "ip" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "mac" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "os" : {
              "properties" : {
                "codename" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "family" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "kernel" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "platform" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "type" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "version" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            }
          }
        },
        "input" : {
          "properties" : {
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "log" : {
          "properties" : {
            "file" : {
              "properties" : {
                "path" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "offset" : {
              "type" : "long"
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "nginx" : {
          "properties" : {
            "access" : {
              "properties" : {
                "agent" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "body_sent" : {
                  "properties" : {
                    "bytes" : {
                      "type" : "long"
                    }
                  }
                },
                "http_version" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "method" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "referrer" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "remote_ip_list" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "response_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "timestamp" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "url" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "user" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "user_name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "useragent" : {
                  "properties" : {
                    "device" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "major" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "minor" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "name" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "os" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "os_full" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "os_major" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "os_minor" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "os_name" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "os_patch" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "os_version" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "patch" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    },
                    "version" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        },
        "tags" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}
'

Logstash settings are as in #11.

Thanks to @leandrojmp for the help.

Hope this helps someone.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.