Creating geoip data for internal networks

Elastic Stack Logstash
3

After a long time away from this issue, I've managed to get the right setup.

In my logstash config, after I've called geoip {} I look for my IP address ranges and manually repopulate the various fields. I'm effectively declaring a new country.

if [srcip] =~ /^139\.80\./ or [srcip] =~ /^10\./ {
   mutate { replace      => { "[geoip][timezone]"      => "Pacific/Auckland" } }
   mutate { replace      => { "[geoip][country_name]"  => "University of Otago" } }
   mutate { replace      => { "[geoip][country_code2]" => "UO" } }
   mutate { replace      => { "[geoip][country_code3]" => "UoO" } }
   mutate { remove_field => [ "[geoip][location]" ] }
   mutate { add_field    => { "[geoip][location]"      => "170.525" } }
   mutate { add_field    => { "[geoip][location]"      => "-45.865" } }
   mutate { convert      => [ "[geoip][location]",        "float" ] }
   mutate { replace      => [ "[geoip][latitude]",        -45.856 ] }
   mutate { convert      => [ "[geoip][latitude]",        "float" ] }
   mutate { replace      => [ "[geoip][longitude]",       170.525 ] }
   mutate { convert      => [ "[geoip][longitude]",       "float" ] }
}

To help keep me honest (actually, to make sure that all my settings are visible in the configuration) I explicitly set a template when writing to ES, rather than rely on state that's already there.

elasticsearch { embedded => "false"
  cluster => "myclustername"
  protocol => "transport"
  host => "indexing host"
  index => "test" # index name must be in lowercase!
  template => "/etc/logstash/template.d/test"
  template_name => "test"
  template_overwrite =>  "true"
}

(Every time I drop this index, the new template will be used to re-create it.)

For my purposes, I'm using the template to switch off string analyzing (most of the fields are log data and analysis doesn't help), and to make sure I get geo_points done properly. Here's the full template from the dev box :-

{
  "order" : 0,
  "template" : "test*",
  "settings" : { "index.refresh_interval" : "5s" },
  "mappings" : {
    "_default_" : {
      "dynamic_templates" : [
        {
          "message_field" : {
            "mapping" : { "index" : "analyzed", "omit_norms" : true, "type" : "string" },
            "match_mapping_type" : "string",
            "match" : "message" }
          },
        {
          "string_fields" : {
            "mapping" : { "index" : "not_analyzed", "ignore_above" : 256, "type" : "string" },
            "match_mapping_type" : "string",
            "match" : "*" }
          }
       ],
      "properties" : {
        "geoip" : {
          "dynamic" : true,
          "path" : "full",
          "properties" : { "location" : { "type" : "geo_point" } },
          "type" : "object" },
        "@version" : { "index" : "not_analyzed", "type" : "string" }
      },
      "_all" : { "enabled" : true }
    }
  },
  "aliases" : { }
}

The end result is a custom country, with a geo-location that separates my network from the users in the same city (although the point on the map when you zoom in isn't as accurate as I want it to be, I don't think I'll worry about that!)

6 Likes