GeoIP filter missing some ECS fields

[roman-tasi]

I am using the GeoIP Logstash filter and it seems to not have some desired fields for example [mmdb][isp]. Overall it has no as or mmdb fields, as well as some other random fields. It does have all the geo fields however.

Any idea how to get these missing fields?

[Rios]

ECS doesn't have those fields. Try with ecs_compatibility => "disabled"

You can set it manually:

      mutate {
        add_field => { "[geo][mmdb][isp]" => "%{somevalue}" }
      }

[roman-tasi]

Hi, where do I put the ecs_compatibility => "disabled" in the config? I added it in the code block appropriate to where I want it applied, but when I reload the Logstash service it causes a syntax error.

Edit: Nvm figured out it goes in the geoip{} block

[roman-tasi]

One more question though. If I set it manually won't it not be from the MaxMind database? I want it to be from the MaxMind database, not set manually right?

[Rios]

Yes, put it inside the geoip plugin. Also is an option to put in logstash.yml or pipeline.yml as pipeline.ecs_compatibility.

 geoip {
       source => "[geo][ip]"
       ecs_compatibility => "disabled" # default is from logstash.yml or pipeline.yml
       #default_database_type => "City" # or ASN
       #database => "/etc/logstash/GeoLiteCity.dat"
       #tag_on_failure => ["IP lookup failed"]
   }

If MaxMind provide mmdb fileds no need to set, but that is not the ECS structure, so you need to avoid ECS, which you figure out how.
You can also leave MaxMind to fill, then check if mmdb is missing/empty, add a value like default value.

[roman-tasi]

Hi, I am currently running this:

mutate {
    add_field => {
        "[mmdb][isp]" => "%{somevalue}"
    }
}

because if I don't add the field the mmdb.isp isn't in Kibana at all. However once I add that mutate the mmdb.isp is there, but all the values are just %{somevalue}

Basically is it possible to actually have the real, correct values that are in the MaxMind database? What do you recommend?

[Rios]

If values exist in MaxMind should be visible with ecs_compatibility => "disabled". If you set to always add_field, the value will be overwritten. As I said, you can also leave geoip to fill MaxMind by the plugin, then check if mmdb is missing/empty, add a value like default value for all empty values.

Most likely free GeoLite2-City doesn't have it. The commercial version has more data. Test online demo for GeoIP2 City Plus Database and check info about GeoIP2 ISP Database.

The GeoIP2 City database is a more accurate version of our free GeoLite2 City database.

The default database is GeoLite2-City . This plugin supports several free databases (GeoLite2-City , GeoLite2-Country , GeoLite2-ASN ) and a selection of commercially-licensed databases (GeoIP2-City , GeoIP2-ISP , GeoIP2-Country ).

[Rios]

One more thing, these are lookup values for 8.8.8.8 - Google DNS server. If is not in GeoLite2-City database, it's small chance to other ISP vales exist. So, there are option:
a) manually add, it is not much geo locations, like company offices
b) use commercial version GeoIP2

# ecs_compatibility => "disabled"

         "geoip" => {
           "region_name" => "California",
           "region_code" => "CA",
             "city_name" => "Los Angeles",
             "longitude" => -118.2441,
          "country_name" => "United States",
           "postal_code" => "90009",
              "latitude" => 34.0544,
                    "ip" => "8.8.8.8",
              "location" => {
            "lat" => 34.0544,
            "lon" => -118.2441
        },
        "continent_code" => "NA",
         "country_code2" => "US",
         "country_code3" => "US",
              "dma_code" => 803,
              "timezone" => "America/Los_Angeles"
    }

#ecs_compatibility => "v8"
         "geo" => {
                "country_name" => "United States",
                 "region_name" => "California",
                    "location" => {
                "lon" => -118.2441,
                "lat" => 34.0544
            },
                   "city_name" => "Los Angeles",
             "region_iso_code" => "US-CA",
                 "postal_code" => "90009",
              "continent_code" => "NA",
            "country_iso_code" => "US",
                    "timezone" => "America/Los_Angeles"
        }