Index template for multi-level field


(Florin Andrei) #1

A while ago, I made a Logstash filter that extracts geolocation information from the client IP in Nginx logs, like this:

geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}

Then I had to modify the Filebeat index template that was loaded into ElasticSearch, in order to mark the geoip.location field with the type geo_point. The index template I've used is this:

{
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": true,
        "norms": {
          "enabled": false
        }
      },
      "dynamic_templates": [
        {
          "template1": {
            "mapping": {
              "doc_values": true,
              "ignore_above": 1024,
              "index": "not_analyzed",
              "type": "{dynamic_type}"
            },
            "match": "*"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "type": "string",
          "index": "analyzed"
        },
        "offset": {
          "type": "long",
          "doc_values": "true"
        },
        "geoip"  : {
          "type" : "object",
          "dynamic": true,
          "properties" : {
            "location" : { "type" : "geo_point" }
          }
        }
      }
    }
  },
  "settings": {
    "index.refresh_interval": "5s"
  },
  "template": "filebeat-*"
}

More recently I've used the json plugin in logstash to extract a JSON document into a field, from a different log source, like this:

json {
source => "data"
target => "node_post_data_json"
}

This has created the field node_post_data_json.geocode.coordinates, which I would like to mark as type geo_point. But the problem is, the geo_point field is three levels down, instead of two, and I'm not sure what is the correct syntax in the template for that. I've tried the template shown below, but it doesn't seem to do anything. What is the correct syntax?

{
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": true,
        "norms": {
          "enabled": false
        }
      },
      "dynamic_templates": [
        {
          "template1": {
            "mapping": {
              "doc_values": true,
              "ignore_above": 1024,
              "index": "not_analyzed",
              "type": "{dynamic_type}"
            },
            "match": "*"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "type": "string",
          "index": "analyzed"
        },
        "offset": {
          "type": "long",
          "doc_values": "true"
        },
        "geoip"  : {
          "type" : "object",
          "dynamic": true,
          "properties" : {
            "location" : { "type" : "geo_point" }
          }
        },
        "node_post_data_json": {
          "type" : "object",
          "dynamic": true,
          "properties" : {
            "geocode": {
              "type" : "object",
              "dynamic": true,
              "properties" : {
                "coordinates" : { "type" : "geo_point" }
              }
            }
          }
        }
      }
    }
  },
  "settings": {
    "index.refresh_interval": "5s"
  },
  "template": "filebeat-*"
}

(Florin Andrei) #2

Replying to my own question:

Actually, that's the correct syntax. I just had to wait until the next daily index was created, and the field was automagically created with the intended type.


(ruflin) #3

The issue is that mapping of existing indices with data can't be updated, but as one index is created per in your case, every day a new mapping can be used. Thanks for providing all the details and also the answer for others.


(system) #4