How can I change an existing field type using Python?


(Sean Whalen) #1

Hi Everyone,

I have an open source project that uses Elasticsearch and Kibana to visualize DMARC data for almost a year now.

Just recently, a user found a bug where I mapped a field to a long when it should be text (it went unnoticed for so long because the value is almost always "0".

So now I need to find a way to correct this field type in every user's index and Kibana index pattern, preferably in a fully automated way.

My attempt at doing this failed, but hopefully it gives you a good idea of what I'm going for:

def migrate_indexes(aggregate_indexes=None, forensic_indexes=None):
    """
    Updates index mappings

    Args:
        aggregate_indexes (list): A list of aggregate index names
        forensic_indexes (list): A list of forensic index names
    """
    if aggregate_indexes is None:
        aggregate_indexes = []
    if forensic_indexes is None:
        forensic_indexes = []
    for aggregate_index_name in aggregate_indexes:
        aggregate_index = Index(aggregate_index_name)
        body = { "properties": {"published_policy.fo": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
        }
        }
        doc = "doc"
        fo_field = "published_policy.fo"
        fo = "fo"
        fo_mapping = aggregate_index.get_field_mapping(fields=[fo_field])[
            aggregate_index_name]["mappings"][doc][fo_field]["mapping"][fo]
        fo_type = fo_mapping["type"]
        if fo_type == "long":
            aggregate_index.put_mapping(doc_type=doc, body=body)
    for forensic_index in forensic_indexes:
        pass

elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'mapper [published_policy.fo] of different type, current_type [long], merged_type [text]')

What am I doing wrong, and how can I fix this mess?


(Mark Walkom) #2

You cannot change a mapping once it is in place.
Your best option would be to use a template and then update that, so future indices use the correct format.


(Sean Whalen) #3

Thanks for the very fast reply. Is there any way to copy the existing data over to a new index with the correct mapping, so I don't lose data?


(Mark Walkom) #4

You can do a reindex into a new index, then delete the old index and put an alias on the new index so it can still be queried via the original name.