Mappings types

Hi Elastic community,

I'm having a hard time figuring out what is wrong with my index template configuration, but Kibana shows me a message of "mapping types for this template uses types, which have been removed". I don't know what I did wrong here. I've looked for this page, but still I don't what to do. This is not an old template from an older ES version. I'm currently using 8.1. Here's the definition of the mapping in the template:

{
  "mappings": {
    "dynamic": "true",
    "_data_stream_timestamp": {
      "enabled": true
    },
    "_size": {
      "enabled": false
    },
    "dynamic_templates": [
      {
        "match_ip": {
          "match": "ip",
          "match_mapping_type": "string",
          "mapping": {
            "type": "ip"
          }
        }
      },
      {
        "match_message": {
          "match": "message",
          "match_mapping_type": "string",
          "mapping": {
            "type": "match_only_text"
          }
        }
      },
      {
        "strings_as_keyword": {
          "match_mapping_type": "string",
          "mapping": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      }
    ],
    "date_detection": false,
    "numeric_detection": false,
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "agent": {
        "properties": {
          "ephemeral_id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "hostname": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "data_stream": {
        "properties": {
          "dataset": {
            "type": "constant_keyword",
            "value": "sophos.xdr"
          },
          "namespace": {
            "type": "constant_keyword",
            "value": "default"
          },
          "type": {
            "type": "constant_keyword",
            "value": "logs"
          }
        }
      },
      "ecs": {
        "properties": {
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "elastic_agent": {
        "properties": {
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "snapshot": {
            "type": "boolean"
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "event": {
        "properties": {
          "dataset": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "host": {
        "properties": {
          "architecture": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "containerized": {
            "type": "boolean"
          },
          "hostname": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "id": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "ip": {
            "type": "ip"
          },
          "mac": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "os": {
            "properties": {
              "codename": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "family": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "kernel": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "platform": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "type": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "version": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          }
        }
      },
      "input": {
        "properties": {
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "log": {
        "properties": {
          "file": {
            "properties": {
              "path": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "offset": {
            "type": "long"
          }
        }
      },
      "sophos": {
        "properties": {
          "event": {
            "properties": {
              "category": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "created_at": {
                "type": "date"
              },
              "description": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "endpoint_id": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "endpoint_type": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "id": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "kind": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "logged_user": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "severity": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "type": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          }
        }
      },
      "source": {
        "properties": {
          "domain": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "ip": {
            "type": "ip"
          }
        }
      }
    }
  }
}

Ok I used your and and made it a whole template

It will help in the future if you provide the entire request and the entire error message

tl;dr _size is not a valid template field

Why you got a "types" error is not clear to me

PUT _index_template/discuss-template
{
  "index_patterns": [
    "my-index*",
    "bar-index*"
  ],
  "template": {
    "settings": {
      "number_of_shards": 1
    },
    "mappings": {
      "dynamic": "true",
      "_data_stream_timestamp": {
        "enabled": true
      },
      "_size": {
        "enabled": false
      },
      "dynamic_templates": [
        {
          "match_ip": {
            "match": "ip",
            "match_mapping_type": "string",
            "mapping": {
              "type": "ip"
            }
          }
        },
        {
          "match_message": {
            "match": "message",
            "match_mapping_type": "string",
            "mapping": {
              "type": "match_only_text"
            }
          }
        },
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": false,
      "numeric_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "agent": {
          "properties": {
            "ephemeral_id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "hostname": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "type": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "data_stream": {
          "properties": {
            "dataset": {
              "type": "constant_keyword",
              "value": "sophos.xdr"
            },
            "namespace": {
              "type": "constant_keyword",
              "value": "default"
            },
            "type": {
              "type": "constant_keyword",
              "value": "logs"
            }
          }
        },
        "ecs": {
          "properties": {
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "elastic_agent": {
          "properties": {
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "snapshot": {
              "type": "boolean"
            },
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "event": {
          "properties": {
            "dataset": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "host": {
          "properties": {
            "architecture": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "containerized": {
              "type": "boolean"
            },
            "hostname": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "ip": {
              "type": "ip"
            },
            "mac": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "os": {
              "properties": {
                "codename": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "family": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "kernel": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "platform": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "type": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "version": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            }
          }
        },
        "input": {
          "properties": {
            "type": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "log": {
          "properties": {
            "file": {
              "properties": {
                "path": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "offset": {
              "type": "long"
            }
          }
        },
        "sophos": {
          "properties": {
            "event": {
              "properties": {
                "category": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "created_at": {
                  "type": "date"
                },
                "description": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "endpoint_id": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "endpoint_type": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "id": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "kind": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "logged_user": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "severity": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "type": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            }
          }
        },
        "source": {
          "properties": {
            "domain": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "ip": {
              "type": "ip"
            }
          }
        }
      }
    }
  }
}

Results in

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "composable template [discuss-template] template after composition is invalid"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "composable template [discuss-template] template after composition is invalid",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "invalid composite mappings for [discuss-template]",
      "caused_by" : {
        "type" : "mapper_parsing_exception",
        "reason" : "Failed to parse mapping: Root mapping definition has unsupported parameters:  [_size : {enabled=false}]",
        "caused_by" : {
          "type" : "mapper_parsing_exception",
          "reason" : "Root mapping definition has unsupported parameters:  [_size : {enabled=false}]"
        }
      }
    }
  },
  "status" : 400
}

Take out the size parameter which is not valid...

Remove this

      "_size": {
        "enabled": false
      },

and it works fine.

{
  "acknowledged" : true
}

Well, _size metafield was part of my configuration because I once installed mapper size plugin. Since it is still installed, it work and I don't get any error. The message is displayed in Kibana UI in the Mappings section when I edit that template.

Just in case, I tried to remove _size field in the mapping and it doesn't change anything.

I am not familiar with the mapper size plugin...

_size may be returned by the plugin but it may not mean it is valid to set as a parameter can you share the full json and the command you are using to create the template / mapping.

That mapping type detected error generally indicates you are trying to set the

_type parameter to something other than _doc

or just remove it

Here is the whole command to create the template. Nothing is related to _type or type.

PUT _index_template/logs-sophos.xdr
{
  "index_patterns": [
    "logs-sophos.xdr-*"
  ],
  "priority": 200,
  "composed_of": [],
  "data_stream": {
    "hidden": false,
    "allow_custom_routing": false
  },
  "template": {
    "mappings": {
      "_data_stream_timestamp": {
        "enabled": true
      },
      "numeric_detection": false,
      "dynamic": true,
      "dynamic_templates": [
        {
          "match_ip": {
            "mapping": {
              "type": "ip"
            },
            "match_mapping_type": "string",
            "match": "ip"
          }
        },
        {
          "match_message": {
            "mapping": {
              "type": "match_only_text"
            },
            "match_mapping_type": "string",
            "match": "message"
          }
        },
        {
          "strings_as_keyword": {
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        }
      ],
      "date_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "agent": {
          "properties": {
            "hostname": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "name": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "ephemeral_id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "type": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "version": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "log": {
          "properties": {
            "file": {
              "properties": {
                "path": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            },
            "offset": {
              "type": "long"
            }
          }
        },
        "elastic_agent": {
          "properties": {
            "id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "version": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "snapshot": {
              "type": "boolean"
            }
          }
        },
        "source": {
          "properties": {
            "domain": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "ip": {
              "type": "ip"
            }
          }
        },
        "input": {
          "properties": {
            "type": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "ecs": {
          "properties": {
            "version": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "data_stream": {
          "properties": {
            "namespace": {
              "type": "constant_keyword",
              "value": "default"
            },
            "type": {
              "type": "constant_keyword",
              "value": "logs"
            },
            "dataset": {
              "type": "constant_keyword",
              "value": "sophos.xdr"
            }
          }
        },
        "sophos": {
          "properties": {
            "event": {
              "properties": {
                "type": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "endpoint_type": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "endpoint_id": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "id": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "category": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "description": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "kind": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "created_at": {
                  "type": "date"
                },
                "logged_user": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "severity": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            }
          }
        },
        "host": {
          "properties": {
            "hostname": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "os": {
              "properties": {
                "kernel": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "codename": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "name": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "family": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "type": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "version": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "platform": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            },
            "containerized": {
              "type": "boolean"
            },
            "ip": {
              "type": "ip"
            },
            "name": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "mac": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "architecture": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "event": {
          "properties": {
            "dataset": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

I just ran that exact command on my 8.1.2 and it worked fine result.

Perhaps I am missing some nuance, or it is simply related to the mapper size plugin which I do not have installed.

{
  "acknowledged" : true
}

Exactly what error did you get when you ran that?

This returned the correct results

GET _index_template/logs-sophos.xdr

And it shows up in the Template Management Fine as well

No error is returned when running the PUT command to update the template nor to show it with a GET. The warning/error comes from Kibana.

Update I have now installed the mapper size plugin and I do not see an issue either and it seems to work as designed with the _size enabled

PUT _index_template/logs-sophos.xdr
{
  "index_patterns": [
    "logs-sophos.xdr-*"
  ],
  "priority": 200,
  "composed_of": [],
  "data_stream": {
    "hidden": false,
    "allow_custom_routing": false
  },
  "template": {
    "mappings": {
      "_data_stream_timestamp": {
        "enabled": true
      },
      "_size": {
        "enabled": true
      },
      "numeric_detection": false,
      "dynamic": true,
      "dynamic_templates": [
        {
          "match_ip": {
            "mapping": {
              "type": "ip"
            },
            "match_mapping_type": "string",
            "match": "ip"
          }
        },
        {
          "match_message": {
            "mapping": {
              "type": "match_only_text"
            },
            "match_mapping_type": "string",
            "match": "message"
          }
        },
        {
          "strings_as_keyword": {
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        }
      ],
      "date_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "agent": {
          "properties": {
            "hostname": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "name": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "ephemeral_id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "type": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "version": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "log": {
          "properties": {
            "file": {
              "properties": {
                "path": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            },
            "offset": {
              "type": "long"
            }
          }
        },
        "elastic_agent": {
          "properties": {
            "id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "version": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "snapshot": {
              "type": "boolean"
            }
          }
        },
        "source": {
          "properties": {
            "domain": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "ip": {
              "type": "ip"
            }
          }
        },
        "input": {
          "properties": {
            "type": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "ecs": {
          "properties": {
            "version": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "data_stream": {
          "properties": {
            "namespace": {
              "type": "constant_keyword",
              "value": "default"
            },
            "type": {
              "type": "constant_keyword",
              "value": "logs"
            },
            "dataset": {
              "type": "constant_keyword",
              "value": "sophos.xdr"
            }
          }
        },
        "sophos": {
          "properties": {
            "event": {
              "properties": {
                "type": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "endpoint_type": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "endpoint_id": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "id": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "category": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "description": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "kind": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "created_at": {
                  "type": "date"
                },
                "logged_user": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "severity": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            }
          }
        },
        "host": {
          "properties": {
            "hostname": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "os": {
              "properties": {
                "kernel": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "codename": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "name": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "family": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "type": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "version": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "platform": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            },
            "containerized": {
              "type": "boolean"
            },
            "ip": {
              "type": "ip"
            },
            "name": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "mac": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "architecture": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "event": {
          "properties": {
            "dataset": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

results

{
  "acknowledged" : true
}

Put in a doc

POST logs-sophos.xdr-test/_doc
{
          "@timestamp": "2022-05-11T17:30:26.000Z"
  
}

Search it and get results

GET .ds-logs-sophos.xdr-test-2022.05.12-000001/_search
{
  "fields": [
    "_size"
  ]
}

results

{
  "took" : 533,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".ds-logs-sophos.xdr-test-2022.05.12-000001",
        "_id" : "9E6LuIABpvjF7kd-uhzW",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2022-05-11T17:30:26.000Z"
        },
        "fields" : {
          "_size" : [
            58
          ]
        }
      }
    ]
  }
}

Ohh I just saw you are using UI... I will take a look but through API all seems good

I have reproduced ... :slight_smile:

So it looks like you are loading the template via the API and then going to the UI... and stepping through... I get the same error

The offending code appears to be this.

I would need to look what _data_stream_timestamp is.. I do not notice it in any of my OOTB templates not sure where you got it from...

"_data_stream_timestamp": {
        "enabled": true
      }

If I take it out and LOAD via the API it seems to work... if I leave it in I get the same error..

Also I went through the UI to try to create the mapping and tried to load the mappings via JSON and this is what I got

if I take out

"_data_stream_timestamp": {
        "enabled": true
      }

It loads Fine

Looks like it may be a legacy setting used for migrations from indices to data streams I do not think it is needed now for new data streams I can not find it in the reference docs

Found a reference here

Wow, you're a genius! Seriously, I don't know how I end up with that setting, but it might come from different manipulations I've done without looking thoroughly.

Thank you for your help!

Philippe

1 Like