Migrating from ES 5 to ES 8, problems in index

Anibal_Ardid · January 18, 2023, 7:59pm

Hi !
I have this json that i use to index

{
    "settings":{
        "number_of_shards":1,
        "number_of_replicas":0,
        "index.mapping.total_fields.limit":100000,
        "analysis":{
            "filter":{
                "partial_filter":{
                    "type":"edge_ngram",
                    "min_gram":2,
                    "max_gram":20,
                    "token_chars":[
                        "letter",
                        "digit"
                    ]
                }
            },
            "analyzer":{
                "partial-match":{
                    "type":"custom",
                    "tokenizer":"standard",
                    "filter":[
                        "lowercase",
                        "asciifolding",
                        "partial_filter"
                    ]
                },
                "exact-match":{
                    "type":"custom",
                    "tokenizer":"standard",
                    "filter":[
                        "lowercase",
                        "asciifolding"
                    ]
                },
                "last-name":{
                    "type":"custom",
                    "tokenizer":"keyword",
                    "filter":[
                        "lowercase",
                        "asciifolding"
                    ]
                }
            }
        }
    },
    "mappings":{
        "_default_":{
            "_all":{
                "enabled" : true,
                "type" : "string",
                "analyzer" : "partial-match",
                "search_analyzer" : "standard"
            },
            "dynamic_templates":[
                {
                    "string_template":{
                        "match":"*",
                        "match_mapping_type":"string",
                        "mapping":{
                            "type":"string",
                            "analyzer" : "partial-match",
                            "search_analyzer" : "standard",
                            "fields":{
                                "{name}":{
                                    "type":"text",
                                    "index":"analyzed",
                                    "analyzer":"partial-match",
                                    "search_analyzer":"standard"
                                },
                                "exact-match":{
                                    "type":"string",
                                    "index":"analyzed",
                                    "analyzer":"exact-match",
                                    "search_analyzer":"exact-match"
                                }
                            }
                        }
                    }
                }
            ],
            "properties":{
                "post_title":{
                    "type":"text",
                    "analyzer" : "partial-match",
                    "search_analyzer" : "standard",
                    "fields":{
                        "sort-order":{
                            "type":"string",
                            "fielddata": true,
                            "index":"analyzed",
                            "analyzer":"last-name",
                            "search_analyzer":"last-name"
                        }
                    }
                },
                "date":{
                    "type":"date",
                    "format":"yyyy-MM-dd HH:mm:ss"
                },
                "orderby_date":{
                    "type":"date",
                    "format":"yyyy-MM-dd HH:mm:ss"
                },
                "post_status":{
                    "type":"string",
                    "index":"not_analyzed"
                },
                "visible":{
                    "type":"boolean",
                    "index":"not_analyzed"
                },
                "people":{
                    "type":"nested",
                    "properties":{
                        "id":{
                            "type":"integer",
                            "index":"not_analyzed"
                        },
                        "post_title":{
                            "type":"string",
                            "index":"not_analyzed"
                        }
                    }
                },
                "practice_area":{
                    "type":"nested",
                    "properties":{
                        "id":{
                            "type":"integer",
                            "index":"not_analyzed"
                        },
                        "post_title":{
                            "type":"string",
                            "index":"not_analyzed"
                        }
                    }
                },
                "industry":{
                   "type":"nested",
                    "properties":{
                        "id":{
                            "type":"integer",
                            "index":"not_analyzed"
                        },
                        "post_title":{
                            "type":"string",
                            "index":"not_analyzed"
                        }
                    }
                },
                "news_item_type":{
                    "type":"nested",
                    "properties":{
                        "id":{
                            "type":"integer",
                            "index":"not_analyzed"
                        },
                        "term":{
                            "type":"string",
                            "index":"not_analyzed"
                        }
                    }
                },
                "publication_type":{
                    "type":"nested",
                    "properties":{
                        "id":{
                            "type":"integer",
                            "index":"not_analyzed"
                        },
                        "term":{
                            "type":"string",
                            "index":"not_analyzed"
                        }
                    }
                },
                "position":{
                    "type":"nested",
                    "properties":{
                        "id":{
                            "type":"integer",
                            "index":"not_analyzed"
                        },
                        "term":{
                            "type":"string",
                            "index":"not_analyzed"
                        }
                    }
                },
                "office_location":{
                    "type":"nested",
                    "properties":{
                        "id":{
                            "type":"integer",
                            "index":"not_analyzed"
                        },
                        "post_title":{
                            "type":"string",
                            "index":"not_analyzed"
                        }
                    }
                },
                "bar_memberships_fieldset":{
                    "type":"nested",
                    "properties":{
                        "bar_admission":{
                            "type":"nested",
                            "properties":{
                                "id":{
                                    "type":"integer",
                                    "index":"not_analyzed"
                                },
                                "term":{
                                    "type":"string",
                                    "index":"not_analyzed"
                                }
                            }
                        }
                    }
                },
                "education_fieldset":{
                    "type":"nested",
                    "properties":{
                        "law_school":{
                            "type":"string",
                            "index":"not_analyzed"
                        },
                        "school":{
                            "type":"nested",
                            "properties":{
                                "id":{
                                    "type":"integer",
                                    "index":"not_analyzed"
                                },
                                "term":{
                                    "type":"string",
                                    "index":"not_analyzed"
                                }
                            }
                        }
                    }
                },
                "languages_fieldset":{
                    "type":"nested",
                    "properties":{
                        "id":{
                            "type":"integer",
                            "index":"not_analyzed"
                        },
                        "term":{
                            "type":"string",
                            "index":"not_analyzed"
                        }
                    }
                },
                "parent_blog":{
                    "type": "nested",
                    "properties":{
                        "id":{
                            "type":"integer",
                            "index":"not_analyzed"
                        }
                    }    
                }
            }
        },
        "person":{
            "properties":{
                "last_name":{
                    "type":"text",
                    "analyzer" : "partial-match",
                    "search_analyzer" : "standard",
                    "fields":{
                        "raw":{
                            "type":"string",
                            "index":"not_analyzed"
                        },
                        "sort-order":{
                            "type":"string",
                            "fielddata": true,
                            "index":"analyzed",
                            "analyzer":"last-name",
                            "search_analyzer":"last-name"
                        },
                        "exact-match":{
                            "type":"string",
                            "index":"analyzed",
                            "analyzer":"exact-match",
                            "search_analyzer":"exact-match"
                        }
                    }
                },
                "first_name":{
                    "type":"text",
                    "analyzer" : "partial-match",
                    "search_analyzer" : "standard",
                    "fields":{
                        "raw":{
                            "type":"string",
                            "index":"not_analyzed"
                        },
                        "sort-order":{
                            "type":"string",
                            "fielddata": true,
                            "index":"analyzed",
                            "analyzer":"last-name",
                            "search_analyzer":"last-name"
                        },
                        "exact-match":{
                            "type":"string",
                            "index":"analyzed",
                            "analyzer":"exact-match",
                            "search_analyzer":"exact-match"
                        }
                    }
                }
            }
        }
    }
}

I changed to (removing default, not_analyzed,changing string to keyword):

{
    "settings":{
        "number_of_shards":1,
        "number_of_replicas":0,
        "index.mapping.total_fields.limit":100000,
        "analysis":{
            "filter":{
                "partial_filter":{
                    "type":"edge_ngram",
                    "min_gram":2,
                    "max_gram":20,
                    "token_chars":[
                        "letter",
                        "digit"
                    ]
                }
            },
            "analyzer":{
                "partial-match":{
                    "type":"custom",
                    "tokenizer":"standard",
                    "filter":[
                        "lowercase",
                        "asciifolding",
                        "partial_filter"
                    ]
                },
                "exact-match":{
                    "type":"custom",
                    "tokenizer":"standard",
                    "filter":[
                        "lowercase",
                        "asciifolding"
                    ]
                },
                "last-name":{
                    "type":"custom",
                    "tokenizer":"keyword",
                    "filter":[
                        "lowercase",
                        "asciifolding"
                    ]
                }
            }
        }
    },
    "mappings":{
        "_all":{
            "enabled" : true,
            "type" : "string",
            "analyzer" : "partial-match",
            "search_analyzer" : "standard"
        },
        "dynamic_templates":[
            {
                "string_template":{
                    "match":"*",
                    "match_mapping_type":"string",
                    "mapping":{
                        "type":"keyword",
                        "analyzer" : "partial-match",
                        "search_analyzer" : "standard",
                        "fields":{
                            "{name}":{
                                "type":"text",
                                "index":"analyzed",
                                "analyzer":"partial-match",
                                "search_analyzer":"standard"
                            },
                            "exact-match":{
                                "type":"keyword",
                                "index":"analyzed",
                                "analyzer":"exact-match",
                                "search_analyzer":"exact-match"
                            }
                        }
                    }
                }
            }
        ],
        "properties":{
            "post_title":{
                "type":"text",
                "analyzer" : "partial-match",
                "search_analyzer" : "standard",
                "fields":{
                    "sort-order":{
                        "type":"keyword",
                        "fielddata": true,
                        "index":"analyzed",
                        "analyzer":"last-name",
                        "search_analyzer":"last-name"
                    }
                }
            },
            "date":{
                "type":"date",
                "format":"yyyy-MM-dd HH:mm:ss"
            },
            "orderby_date":{
                "type":"date",
                "format":"yyyy-MM-dd HH:mm:ss"
            },
            "post_status":{
                "type":"keyword"
            },
            "visible":{
                "type":"boolean"
            },
            "people":{
                "type":"nested",
                "properties":{
                    "id":{
                        "type":"integer"
                    },
                    "post_title":{
                        "type":"keyword"
                    }
                }
            },
            "practice_area":{
                "type":"nested",
                "properties":{
                    "id":{
                        "type":"integer"
                    },
                    "post_title":{
                        "type":"keyword"
                    }
                }
            },
            "industry":{
               "type":"nested",
                "properties":{
                    "id":{
                        "type":"integer"
                    },
                    "post_title":{
                        "type":"keyword"
                    }
                }
            },
            "news_item_type":{
                "type":"nested",
                "properties":{
                    "id":{
                        "type":"integer"
                    },
                    "term":{
                        "type":"keyword"
                    }
                }
            },
            "publication_type":{
                "type":"nested",
                "properties":{
                    "id":{
                        "type":"integer"
                    },
                    "term":{
                        "type":"keyword"
                    }
                }
            },
            "position":{
                "type":"nested",
                "properties":{
                    "id":{
                        "type":"integer"
                    },
                    "term":{
                        "type":"keyword"
                    }
                }
            },
            "office_location":{
                "type":"nested",
                "properties":{
                    "id":{
                        "type":"integer"
                    },
                    "post_title":{
                        "type":"keyword"
                    }
                }
            },
            "bar_memberships_fieldset":{
                "type":"nested",
                "properties":{
                    "bar_admission":{
                        "type":"nested",
                        "properties":{
                            "id":{
                                "type":"integer"
                            },
                            "term":{
                                "type":"keyword"
                            }
                        }
                    }
                }
            },
            "education_fieldset":{
                "type":"nested",
                "properties":{
                    "law_school":{
                        "type":"keyword"
                    },
                    "school":{
                        "type":"nested",
                        "properties":{
                            "id":{
                                "type":"integer"
                            },
                            "term":{
                                "type":"keyword"
                            }
                        }
                    }
                }
            },
            "languages_fieldset":{
                "type":"nested",
                "properties":{
                    "id":{
                        "type":"integer"
                    },
                    "term":{
                        "type":"keyword"
                    }
                }
            },
            "parent_blog":{
                "type": "nested",
                "properties":{
                    "id":{
                        "type":"integer"
                    }
                }    
            }
        },
        "person":{
            "properties":{
                "last_name":{
                    "type":"text",
                    "analyzer" : "partial-match",
                    "search_analyzer" : "standard",
                    "fields":{
                        "raw":{
                            "type":"keyword"
                        },
                        "sort-order":{
                            "type":"keyword",
                            "fielddata": true,
                            "index":"analyzed",
                            "analyzer":"last-name",
                            "search_analyzer":"last-name"
                        },
                        "exact-match":{
                            "type":"keyword",
                            "index":"analyzed",
                            "analyzer":"exact-match",
                            "search_analyzer":"exact-match"
                        }
                    }
                },
                "first_name":{
                    "type":"text",
                    "analyzer" : "partial-match",
                    "search_analyzer" : "standard",
                    "fields":{
                        "raw":{
                            "type":"keyword"
                        },
                        "sort-order":{
                            "type":"keyword",
                            "fielddata": true,
                            "index":"analyzed",
                            "analyzer":"last-name",
                            "search_analyzer":"last-name"
                        },
                        "exact-match":{
                            "type":"keyword",
                            "index":"analyzed",
                            "analyzer":"exact-match",
                            "search_analyzer":"exact-match"
                        }
                    }
                }
            }
        }
    }
}

And I have this error:

  [Elasticsearch\Common\Exceptions\BadRequest400Exception]
  {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"unknown parameter [search_analyzer] on mapper [sort-order] of type [keyword]"}],"type":"mapper_parsing_e
  xception","reason":"Failed to parse mapping: unknown parameter [search_analyzer] on mapper [sort-order] of type [keyword]","caused_by":{"type":"mapper_parsing_exception","re
  ason":"unknown parameter [search_analyzer] on mapper [sort-order] of type [keyword]"}},"status":400}

I'm using the latest elasticsearch php composer library.

Thanks in advance

Anibal_Ardid · January 18, 2023, 8:01pm

I tried also without _all_

Anibal_Ardid · January 24, 2023, 6:57pm

nothing knows about it ?

leandrojmp · January 24, 2023, 8:20pm

The format of the templates changed in version 8, you need to adapt your templates according to the documentation.

From version 5 to version 8 a lot of things changed, you will need to check every breaking change from your current version 5 until the last 8 version to see what you will need to change.

Anibal_Ardid · January 26, 2023, 2:17pm

Hola Leandro !
Hi !
But, this change a lot of things, i checked and changed some, but i have fails that i couldnt recognize where is the problem

system · February 23, 2023, 2:18pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Need help with ES Query Elasticsearch	1	316	July 6, 2017
Document(s) failed to index \| mapper_parsing_exception Elasticsearch	3	1061	November 13, 2022
Index configurations file (mappings & settings) in ES 1.7.x Elasticsearch	4	893	July 5, 2017
Problem with JSON query understanding Elasticsearch	1	301	April 19, 2021
Issue while indexing with custom analyzers Elasticsearch	3	23	July 17, 2024

Migrating from ES 5 to ES 8, problems in index

Related topics