How can I change an existing field type using Python?

Hi Everyone,

I have an open source project that uses Elasticsearch and Kibana to visualize DMARC data for almost a year now.

Just recently, a user found a bug where I mapped a field to a long when it should be text (it went unnoticed for so long because the value is almost always "0".

So now I need to find a way to correct this field type in every user's index and Kibana index pattern, preferably in a fully automated way.

My attempt at doing this failed, but hopefully it gives you a good idea of what I'm going for:

def migrate_indexes(aggregate_indexes=None, forensic_indexes=None):
    """
    Updates index mappings

    Args:
        aggregate_indexes (list): A list of aggregate index names
        forensic_indexes (list): A list of forensic index names
    """
    if aggregate_indexes is None:
        aggregate_indexes = []
    if forensic_indexes is None:
        forensic_indexes = []
    for aggregate_index_name in aggregate_indexes:
        aggregate_index = Index(aggregate_index_name)
        body = { "properties": {"published_policy.fo": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
        }
        }
        doc = "doc"
        fo_field = "published_policy.fo"
        fo = "fo"
        fo_mapping = aggregate_index.get_field_mapping(fields=[fo_field])[
            aggregate_index_name]["mappings"][doc][fo_field]["mapping"][fo]
        fo_type = fo_mapping["type"]
        if fo_type == "long":
            aggregate_index.put_mapping(doc_type=doc, body=body)
    for forensic_index in forensic_indexes:
        pass

elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'mapper [published_policy.fo] of different type, current_type [long], merged_type [text]')

What am I doing wrong, and how can I fix this mess?

You cannot change a mapping once it is in place.
Your best option would be to use a template and then update that, so future indices use the correct format.

Thanks for the very fast reply. Is there any way to copy the existing data over to a new index with the correct mapping, so I don't lose data?

You can do a reindex into a new index, then delete the old index and put an alias on the new index so it can still be queried via the original name.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.