How do i reindex data to resolve type conflict?

Elastic Stack Elasticsearch
1

Can someone provide a full concise example of how I would reindex my data to resolve a conflict? I have two fields where the type changed from string to integer. I don't understand how the reindex process works or why I even need to do it. Is there a full guide that explains how to do this somewhere?

2

After a lot of banging my head on the keyboard I was able to resolve this using these steps:

  1. determine the indexes that need to be adjusted: the following python code will filter all indexes containing the fields you specify as well as the differences between the types for each index.

    #!/usr/bin/python

    import elasticsearch

    es = elasticsearch.Elasticsearch('http://localhost:9200')
    index_list = es.indices.get('*')

    #the fields in kibana that show as a conflict
    fields = ['field1', 'field2', 'field3']

    #get a list of mappings, filtered by the fields we are inspecting
    mappings = es.indices.get_field_mapping(",".join(fields), "*")

    for m in mappings:
    if mappings[m]['mappings']:
    print m
    print mappings[m]['mappings']

  2. download the broken index
    curl -XGET http://${ELASTICSEARCH_HOST}:9200/${INDEX_NAME}/?pretty > index
    or if you have an index that already has the correct types set, download that instead as you can use that to create your new indexes instead of manually editing the file

  3. edit the downloaded "index" file and remove the 2nd line (the one containing the name of the index) as well as the 2nd to last line (containing the corresponding closing brace). If you had to download a broken index in step 2 because you didn't have one with the correct types, also edit the "index" file to reflect the correct types.

  4. upload the new index file
    curl -XPUT http://${ELASTICSEARCH_HOST}:9200/${INDEX_NAME}-fixed -d@index
    you can run this step multiple times to fix any indexes of the same type that were output in step 1

  5. reindex your data from the old index into the new
    curl -XPOST http://${ELASTICSEARCH_HOST}:9200/_reindex -d '{ "source": { "index" : "${INDEX_NAME}" },"dest" : { "index" : "${INDEX_NAME}-fixed" } }'

  6. delete the old broken index
    curl -XDELETE http://${ELASTICSEARCH_HOST}:9200/${INDEX_NAME}

  7. refresh your field list in kibana

2 Likes
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.