I have an open source project that uses Elasticsearch and Kibana to visualize DMARC data for almost a year now.
Just recently, a user found a bug where I mapped a field to a long when it should be text (it went unnoticed for so long because the value is almost always "0".
So now I need to find a way to correct this field type in every user's index and Kibana index pattern, preferably in a fully automated way.
My attempt at doing this failed, but hopefully it gives you a good idea of what I'm going for:
def migrate_indexes(aggregate_indexes=None, forensic_indexes=None):
"""
Updates index mappings
Args:
aggregate_indexes (list): A list of aggregate index names
forensic_indexes (list): A list of forensic index names
"""
if aggregate_indexes is None:
aggregate_indexes = []
if forensic_indexes is None:
forensic_indexes = []
for aggregate_index_name in aggregate_indexes:
aggregate_index = Index(aggregate_index_name)
body = { "properties": {"published_policy.fo": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
doc = "doc"
fo_field = "published_policy.fo"
fo = "fo"
fo_mapping = aggregate_index.get_field_mapping(fields=[fo_field])[
aggregate_index_name]["mappings"][doc][fo_field]["mapping"][fo]
fo_type = fo_mapping["type"]
if fo_type == "long":
aggregate_index.put_mapping(doc_type=doc, body=body)
for forensic_index in forensic_indexes:
pass
elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'mapper [published_policy.fo] of different type, current_type [long], merged_type [text]')
What am I doing wrong, and how can I fix this mess?
You cannot change a mapping once it is in place.
Your best option would be to use a template and then update that, so future indices use the correct format.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.