Refuse field type change in mapping

Hi

Is it possible to ignore data not passing the mapping, like date type for @timestamp. It messes up my index pattern where 99% is all fine.

Most of my index mappings looks like these, which are fine:
{
"mapping": {
"unitylog": {
"properties": {
"@timestamp": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss.SSS"
},
"Component": {
"type": "text"
},
"Details": {
"type": "text"
},
"Level": {
"type": "integer"
},
"OperationId": {
"type": "text"
},
"SystemLogId": {
"type": "integer"
},
"TenantId": {
"type": "text"
},
"host": {
"type": "text"
},
"message": {
"type": "text"
}
}
}
}
}

But a few indexes looks like these, I suspect it being some illegal data:
{
"mapping": {
"unitylog": {
"properties": {
"@timestamp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Component": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Details": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Level": {
"type": "long"
},
"OperationId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"SystemLogId": {
"type": "long"
},
"TenantId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}

Hi,

You can use dynamic template and set that all fields named timestamp must be datetime if it fail it will return error. This way you are sure to have a correct mapping.

Check about the documentation for more details.
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html

Hope it help.

Thanks Dynamic Templating was the keyword, I manage to fix this by firstly specifying no field to be empty and improving the extraction sql.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.