APM GeoIP on Logstash vs Elastic Ingest

[digitalron (Ronald Tumulak)]

Elastic Stack: 7.0.1, Docker Version

We are encountering a problem with the GeoIP filter seemingly having a different behaviour on Logstash versus using the Elasticsearch ingest pipeline. Please see below:

On Logstash:

On Elasticsearch Ingest:

Using the Logstash version generates more data (which we like) but it doesn't seem to generate the country_iso_code field and instead created country_code2, country_code3.

Is this by design or are we missing something?

Below are our filter settings:

Logstash filter:

filter {

    geoip {
      source => "[client][ip]"
      target => "client.geo"
    }

}

Ingest filter:

"processors" : [
      {
        "geoip" : {
          "field": "client.ip",
          "target_field": "client.geo",
          "ignore_missing": true
        }    
      }
    ]

[gil (Gil Raphaelli)]

I don't think you're missing anything. I've opened an issue for handling country_iso_code - as you noted the other fields line up correctly. As I'm sure you've considered, you can use logstash to rename that field.

[digitalron (Ronald Tumulak)]

Thanks Gil! We'll wait for that as we have different deployments, some using Logstash, some none.

Kibana Index Patterns shows the following fields:

which has the field country_iso_code similar to the one used in the Elastic ingest.

Also, our queries result in errors if we use country_code2 or country_code3 instead of country_iso_code, but country_iso_code is all null.

So what we did was to update the template file and include the fields used in the Logstash filter. We also adjusted our query to use country_code2 instead. We're not sure we want to use rename in our Logstash as we want to keep it as simple as possible and stay with our principle of using as few custom filters/rules as possible.

BTW, looking at logs in Logstash, it seems the acceptable fields for the Logstash GeoIP filter are as specified in the sample entry below:

Pipeline aborted due to error {:pipeline_id=>"main", :exception=>java.lang.IllegalArgumentException: illegal field value country_iso_code. valid values are [AUTONOMOUS_SYSTEM_NUMBER, AUTONOMOUS_SYSTEM_ORGANIZATION, CITY_NAME, COUNTRY_NAME, CONTINENT_CODE, CONTINENT_NAME, COUNTRY_CODE2, COUNTRY_CODE3, IP, ISP, POSTAL_CODE, DMA_CODE, REGION_NAME, REGION_CODE, TIMEZONE, LOCATION, LATITUDE, LONGITUDE, ORGANIZATION]