[Solved] Logstash geoip vs. ES 2.x "no dot in fields"

Elastic Stack Logstash
1

Hi there,

we're currently using the following Logstash-Filter for geoip-processing:

geoip {
  source => "client_ip"
  lru_cache_size => 10000
}

This creates for example the following output:

"geoip" => {
  "ip" => "8.8.4.4",
  "country_code2" => "US",
  "country_code3" => "USA",
  "country_name" => "United States",
  "continent_code" => "NA",
  "region_name" => "CA",
  "city_name" => "Mountain View",
  "postal_code" => "94043",
  "latitude" => 37.41919999999999,
  "longitude" => -122.0574,
  "dma_code" => 807,
  "area_code" => 650,
  "timezone" => "America/Los_Angeles",
  "real_region_name" => "California",
  "location" => [
    [0] -122.0574,
    [1] 37.41919999999999
  ]
}

We're still using Logstash 1.5.x with ES 1.7.x but plan to upgrade to ES 2.x.
But here we experience the problem of "dots in fieldnames not supported".

-> https://www.elastic.co/guide/en/logstash/current/_upgrading_logstash_and_elasticsearch_to_2_0.html

-> https://www.elastic.co/guide/en/elasticsearch/reference/2.0/breaking_20_mapping_changes.html#_field_names_may_not_contain_dots

How can we modify the geoip-filter to match the requirements of ES 2.x?

The useragent-filter has "prefix", but "geoip" hasn't.

What are possible solutions?

2

I'm not sure I see the problem. Where are the dots in the fields from the GeoIP filter? Yes, the fields are structured, but there are no actual dots in the fields. Elasticsearch and Kibana may even refer to the nested structure with dots, but again, here there are no dots.

3

Hi,

you're totally right. There is no real problem regarding Logstash.

Our migration plugin complains about geoip:

Dots in field names lead to ambiguous field resolution, in fields:
  _default_:geoip\.location, curator:geoip\.location, ...

But now it really looks like a problem with our index-templates:

{
  "template_1": {
    "order": 0,
    "template": "indexname-*",
    "settings": {},
    "mappings": {
      "_default_": {
        "dynamic_templates": [
          {
            "message_field": {
              "mapping": {
                "index": "analyzed",
                "omit_norms": true,
                "type": "string",
                "fields": {
                  "raw": {
                    "ignore_above": 256,
                    "index": "not_analyzed",
                    "type": "string"
                  }
                }
              },
              "match_mapping_type": "string",
              "match": "message"
            }
          }
        ],
        "properties": {
          "geoip.location": {
            "type": "geo_point",
            "lat_lon": true
          }
        }
      }
    },
    "aliases": {}
  }
}

The mapping of geoip.location is wrong and therefore creating problems with the migration-plugin.

Sorry for bothering you and thank you for pointing me into the right direction!