Refuse field type change in mapping

Elastic Stack Elasticsearch
1

Hi

Is it possible to ignore data not passing the mapping, like date type for @timestamp. It messes up my index pattern where 99% is all fine.

Most of my index mappings looks like these, which are fine:
{
"mapping": {
"unitylog": {
"properties": {
"@timestamp": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss.SSS"
},
"Component": {
"type": "text"
},
"Details": {
"type": "text"
},
"Level": {
"type": "integer"
},
"OperationId": {
"type": "text"
},
"SystemLogId": {
"type": "integer"
},
"TenantId": {
"type": "text"
},
"host": {
"type": "text"
},
"message": {
"type": "text"
}
}
}
}
}

But a few indexes looks like these, I suspect it being some illegal data:
{
"mapping": {
"unitylog": {
"properties": {
"@timestamp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Component": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Details": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Level": {
"type": "long"
},
"OperationId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"SystemLogId": {
"type": "long"
},
"TenantId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}

2

Hi,

You can use dynamic template and set that all fields named timestamp must be datetime if it fail it will return error. This way you are sure to have a correct mapping.

Check about the documentation for more details.
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html

Hope it help.

3

Thanks Dynamic Templating was the keyword, I manage to fix this by firstly specifying no field to be empty and improving the extraction sql.