Ingest pipeline GeoIP processor lacking parent fields on ingestion?

Elastic Stack Elasticsearch
1

In our environment we have custom logs coming in of which a remote IP is ingested to a field called remoteAddr. In our ingest pipeline we're using a GeoIP processor which we use the source field remoteAddr to the target field remoteAddrGeoIP. This then creates the following fields upon ingestion:

remoteAddrGeoIP.city_name
remoteAddrGeoIP.continent_name
remoteAddrGeoIP.country_iso_code
remoteAddrGeoIP.country_name
remoteAddrGeoIP.location.lat
remoteAddrGeoIP.location.lon
remoteAddrGeoIP.region_iso_code
remoteAddrGeoIP.region_name

This almost works as expected with the exception that I'm not sure why the Geo-point field remoteAddrGeoIP.location is "missing". There's a possibility to join the remoteAddrGeoIP.location.lat and remoteAddrGeoIP.location.lon onto the remoteAddrGeoIP.location field, but I'm unsure whether that is the best practice method. I've also tried mapping remoteAddrGeoIP as a nested field with remoteAddrGeoIP.location as a child Geo-point field in the index template without success.

In short:

The remoteAddrGeoIP.location.lat and remoteAddrGeoIP.location.lon fields should be in the remoteAddrGeoIP.location [Geo-point field] as well/instead.

Any ideas?

2

Can you post your .conf file and the these mapping sections within elastic?

3

We're using elastic cloud so I'll share the details as fetched from the dev console:

"mappings" : {
  "_routing" : {
    "required" : false
  },
  "numeric_detection" : false,
  "dynamic_date_formats" : [
    "strict_date_optional_time",
    "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
  ],
  "_source" : {
    "excludes" : [ ],
    "includes" : [ ],
    "enabled" : true
  },
  "dynamic" : true,
  "dynamic_templates" : [ ],
  "date_detection" : true,
  "properties" : {
    "remoteAddrGeoIP" : {
      "type" : "nested",
      "properties" : {
        "remoteAddrGeoIP.location" : {
          "ignore_malformed" : false,
          "type" : "geo_point",
          "ignore_z_value" : true
        }
      }
    },
    "remoteAddr" : {
      "type" : "ip"
    }
  }
}

Pipeline settings:

{
  "testing-pipeline": {
    "processors": [
      {
        "geoip": {
          "field": "remoteAddr",
          "target_field": "remoteAddrGeoIP",
          "ignore_missing": true
        }
      }
    ]
  }
}
4

Your mapping is a bit malformed, here is a cleaned up / more detailed one with an example document.

DELETE discuss-geoip

PUT discuss-geoip
{
  "mappings": {
    "_routing": {
      "required": false
    },
    "numeric_detection": false,
    "dynamic_date_formats": [
      "strict_date_optional_time",
      "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
    ],
    "_source": {
      "excludes": [],
      "includes": [],
      "enabled": true
    },
    "dynamic": true,
    "dynamic_templates": [],
    "date_detection": true,
    "properties": {
      "remoteAddrGeoIP": {
        "properties": {
          "city_name": {
            "type": "keyword"
          },
          "continent_name": {
            "type": "keyword"
          },
          "country_iso_code": {
            "type": "keyword"
          },
          "country_name": {
            "type": "keyword"
          },
          "region_iso_code": {
            "type": "keyword"
          },
          "location": {
            "ignore_malformed": false,
            "type": "geo_point",
            "ignore_z_value": true
          }
        }
      },
      "remoteAddr": {
        "type": "ip"
      }
    }
  }
}

PUT _ingest/pipeline/discuss-geoip
{
  "processors": [
    {
      "geoip": {
        "field": "remoteAddr",
        "target_field": "remoteAddrGeoIP",
        "ignore_missing": true
      }
    }
  ]
}


POST discuss-geoip/_doc/?pipeline=discuss-geoip
{
  "remoteAddr": "47.153.100.100"
}

results

GET discuss-geoip/_search

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "discuss-geoip",
        "_type" : "_doc",
        "_id" : "Dh-WUn0BMB9Z88FB0y-i",
        "_score" : 1.0,
        "_source" : {
          "remoteAddrGeoIP" : {
            "continent_name" : "North America",
            "region_iso_code" : "US-CA",
            "city_name" : "Temecula",
            "country_iso_code" : "US",
            "country_name" : "United States",
            "region_name" : "California",
            "location" : {
              "lon" : -117.1208,
              "lat" : 33.5173
            }
          },
          "remoteAddr" : "47.153.100.100"
        }
      }
    ]
  }
}
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.