Template inheritance: non-dynamic mappings

Elastic Stack Elasticsearch
1

Hi,

Using ES 7.0.0. I've been looking for docs/posts about this, but most info is very old or not clear, and after playing around with it I don't understand how it should work.

I have different indexes that have a common base. There's an index for "cars" and an index for "computers". Cars have a "number_of_wheels", and computers have "number_of_cpus", but both have a "price" etc.

So I create the common template with order 1:

PUT _template/objects
{
  "order" : 1,
  "index_patterns" : [
    "objects-*"
  ],
  "mappings" : {
    "dynamic" : false,
    "properties" : {
      "price" : {
        "type" : "long"
      }
    }
  }
}

And the other 2 templates with order 2, which results in 3 templates:

GET /_template/objects*
{
  "objects-cars" : {
    "order" : 2,
    "index_patterns" : [
      "objects-cars-*"
    ],
    "settings" : { },
    "mappings" : {
      "dynamic" : false,
      "properties" : {
        "number_of_wheels" : {
          "type" : "long"
        }
      }
    },
    "aliases" : { }
  },
  "objects-computers" : {
    "order" : 2,
    "index_patterns" : [
      "objects-computers-*"
    ],
    "settings" : { },
    "mappings" : {
      "dynamic" : false,
      "properties" : {
        "number_of_cpus" : {
          "type" : "long"
        }
      }
    },
    "aliases" : { }
  },
  "objects" : {
    "order" : 1,
    "index_patterns" : [
      "objects-*"
    ],
    "settings" : { },
    "mappings" : {
      "dynamic" : false,
      "properties" : {
        "price" : {
          "type" : "long"
        }
      }
    },
    "aliases" : { }
  }
}

I create a document:

POST /objects-computers-foo/_doc
{
  "price": 15,
  "number_of_cpus": 4
}

Which succeeds:

{
  "_index" : "objects-computers-foo",
  "_type" : "_doc",
  "_id" : "Zc-p420B4UGO-P7XGC0J",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

Seems OK:

GET /objects-computers-foo/_search
{
    "query": {
        "match_all": {}
    }
}

says

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "objects-computers-foo",
        "_type" : "_doc",
        "_id" : "Zc-p420B4UGO-P7XGC0J",
        "_score" : 1.0,
        "_source" : {
          "price" : 15,
          "number_of_cpus" : 4
        }
      }
    ]
  }
}

But I don't expect that I can create a computer with wheels:

POST /objects-computers-foo/_doc
{
  "price": 20,
  "number_of_wheels": 4
}

However, that seem to result in this:

{
  "_index" : "objects-computers-foo",
  "_type" : "_doc",
  "_id" : "h8-r420B4UGO-P7XLzfc",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_seq_no" : 1,
  "_primary_term" : 1
}

The documents are now:

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "objects-computers-foo",
        "_type" : "_doc",
        "_id" : "Zc-p420B4UGO-P7XGC0J",
        "_score" : 1.0,
        "_source" : {
          "price" : 15,
          "number_of_cpus" : 4
        }
      },
      {
        "_index" : "objects-computers-foo",
        "_type" : "_doc",
        "_id" : "h8-r420B4UGO-P7XLzfc",
        "_score" : 1.0,
        "_source" : {
          "price" : 20,
          "number_of_wheels" : 4
        }
      }
    ]
  }
}

Looking at the index, this is what I see:

GET /objects-computers-foo

{
  "objects-computers-foo" : {
    "aliases" : { },
    "mappings" : {
      "dynamic" : "false",
      "properties" : {
        "number_of_cpus" : {
          "type" : "long"
        },
        "price" : {
          "type" : "long"
        }
      }
    },
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "delete_after_365_days"
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "provided_name" : "objects-computers-foo",
        "creation_date" : "1571482572647",
        "priority" : "100",
        "number_of_replicas" : "1",
        "uuid" : "Y-Dt5nosS-i6WohMQeGHnw",
        "version" : {
          "created" : "7000099"
        }
      }
    }
  }
}

Please ignore the ILM of 365 days, that because I have another order 0 template that applies default ILM's of 365 days, but I think it's unrelated.

The index does seem to have a static mapping with only 2 properties: number_of_cpus and price. So why is the number_of_wheels allowed?

Am I not understanding the concept of template inheritance (or dynamic/statis mapping), is this a known issue, is this a bug, or none of the above?

2

Maybe I missed the complete point of static mappings, because this also seems to work:

POST /objects-computers-foo/_doc
{
  "price": 30,
  "number_of_wheels": 6,
  "this_makes_no_sense": "bla"
}

It returns

{
  "_index" : "objects-computers-foo",
  "_type" : "_doc",
  "_id" : "Js-2420B4UGO-P7XKIIZ",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_seq_no" : 2,
  "_primary_term" : 1
}

These are the docs now:

GET /objects-computers-foo/_search
{
    "query": {
        "match_all": {}
    }
}

returns

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "objects-computers-foo",
        "_type" : "_doc",
        "_id" : "Zc-p420B4UGO-P7XGC0J",
        "_score" : 1.0,
        "_source" : {
          "price" : 15,
          "number_of_cpus" : 4
        }
      },
      {
        "_index" : "objects-computers-foo",
        "_type" : "_doc",
        "_id" : "h8-r420B4UGO-P7XLzfc",
        "_score" : 1.0,
        "_source" : {
          "price" : 20,
          "number_of_wheels" : 4
        }
      },
      {
        "_index" : "objects-computers-foo",
        "_type" : "_doc",
        "_id" : "Js-2420B4UGO-P7XKIIZ",
        "_score" : 1.0,
        "_source" : {
          "price" : 30,
          "number_of_wheels" : 6,
          "this_makes_no_sense" : "bla"
        }
      }
    ]
  }
}
3

Looks like it's documented correctly after all, I just made wrong assumptions: Index templates | Elasticsearch Guide [8.11] | Elastic

The key is in the last note:

Setting dynamic to false doesn’t alter the contents of the _source field at all. The _source will still contain the whole JSON document that you indexed. However, any unknown fields will not be added to the mapping and will not be searchable.

I'm still not too sure about how Kibana handles this, since I did see those fields pop-up where I did not expect them to. Will look into that and report back (in case anyone stumbles on the same question).

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.