How can I change an existing field type using Python?

(Sean Whalen) #1

Hi Everyone,

I have an open source project that uses Elasticsearch and Kibana to visualize DMARC data for almost a year now.

Just recently, a user found a bug where I mapped a field to a long when it should be text (it went unnoticed for so long because the value is almost always "0".

So now I need to find a way to correct this field type in every user's index and Kibana index pattern, preferably in a fully automated way.

My attempt at doing this failed, but hopefully it gives you a good idea of what I'm going for:

def migrate_indexes(aggregate_indexes=None, forensic_indexes=None):
    Updates index mappings

        aggregate_indexes (list): A list of aggregate index names
        forensic_indexes (list): A list of forensic index names
    if aggregate_indexes is None:
        aggregate_indexes = []
    if forensic_indexes is None:
        forensic_indexes = []
    for aggregate_index_name in aggregate_indexes:
        aggregate_index = Index(aggregate_index_name)
        body = { "properties": {"": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
        doc = "doc"
        fo_field = ""
        fo = "fo"
        fo_mapping = aggregate_index.get_field_mapping(fields=[fo_field])[
        fo_type = fo_mapping["type"]
        if fo_type == "long":
            aggregate_index.put_mapping(doc_type=doc, body=body)
    for forensic_index in forensic_indexes:

elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'mapper [] of different type, current_type [long], merged_type [text]')

What am I doing wrong, and how can I fix this mess?

(Mark Walkom) #2

You cannot change a mapping once it is in place.
Your best option would be to use a template and then update that, so future indices use the correct format.

(Sean Whalen) #3

Thanks for the very fast reply. Is there any way to copy the existing data over to a new index with the correct mapping, so I don't lose data?

(Mark Walkom) #4

You can do a reindex into a new index, then delete the old index and put an alias on the new index so it can still be queried via the original name.

(system) closed #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.