I want to migrate an index from a cluster to another index in the another cluster but I get error

Elastic Stack Logstash
1

Hi,
I want to migrate an index to another cluster. I can't use reindex because source IP is not whitelisted. also I can't use snapshot
I decided to do this task by writing a logstash pipeline you can see my pipeline below:

input {
    elasticsearch {
        hosts => "${INPUT_ES_HOSTS}"
        index => "${INPUT_INDEX}"
        user => "${INPUT_ES_USERNAME}"
        password => "${INPUT_ES_PASSWORD}"
        docinfo => true
        size => "${INPUT_ES_SIZE:5000}"
        scroll => "${INPUT_ES_SCROLL:5m}"
        ecs_compatibility => disabled
        docinfo_fields => ["_index", "_type", "_id"]
    }
}

output {
    elasticsearch {
        hosts => "${OUTPUT_ES_HOSTS}"
        user => "${OUTPUT_ES_USERNAME}"
        password => "${OUTPUT_ES_PASSWORD}"
        index => "%{[@metadata][_index]}"
        document_id => "%{[@metadata][_id]}"
        document_type => "%{[@metadata][_type]}"
    }
}

when this logstash pipeline starts, I get the following error:

:response=>{"index"=>{"_index"=>"product-definition-v19", "_id"=>"a96bf648-388e-3f67-8f1a-f5dedd029dfd", "status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:3518] object mapping for [tags] tried to parse field [null] as object, but found a concrete value"}}}

tags field is defined as follows:

        "tags" : {
          "properties" : {
            "confidence" : {
              "type" : "double"
            },
            "id" : {
              "type" : "keyword"
            }
          }
        }

And here is a sample of what its value looks like:

    "tags" : [
      {
        "confidence" : 0.932,
        "id" : "a96bf648-388e-3f67-8f1a-f5dedd029dfd"
      }
    ]

I appreciate any help
Thank you.

2

This error means that the mapping of the tags field in the destination index is not the same as the source index.

Before migrating you need to use the same template you used in your source index.

Also, keep in mind that tags is a base ecs field used by logstash and all other tools in the stack, it should not be mapped as an object, but as a keyword as mentioned in the documentation.

Mapping the tags field as an object may cause some issues as logstash uses this field in multiple filters.

3

@leandrojmp Thank you so much

Source and destination index mappings are the same. you can see below
source elastic 7.x

        "tags" : {
          "properties" : {
            "confidence" : {
              "type" : "double"
            },
            "id" : {
              "type" : "keyword"
            }
          }
        }

destination Elastic 8.12

        "tags": {
          "properties": {
            "confidence": {
              "type": "double"
            },
            "id": {
              "type": "keyword"
            }
          }
        }

As you can see they are the same.

I update the output filter like below but no success:

output {
# stdout {codec => "rubydebug"}
    elasticsearch {
        ecs_compatibility => disabled
        hosts => "${OUTPUT_ES_HOSTS}"
        user => "${OUTPUT_ES_USERNAME}"
        password => "${OUTPUT_ES_PASSWORD}"
        index => "%{[@metadata][_index]}"
        document_id => "%{[@metadata][_id]}"
        document_type => "%{[@metadata][_type]}"
    }
}

what do you suggest?

Thank you.

4

Yeah, but it is not what the logstash error is showing, so something is not right, maybe the template was not applied correctly.

Can you share the mapping of the index product-definition-v19 in your destination cluster?

Just use GET product-definition-v19/_mapping on Kibana Dev Tools and share the result.

Also, share a sample document of the same index, use the following query:

GET product-definition-v19/_search
{
  "size": 1,
  "query": {
    "exists": {
      "field": "tags"
    }
  }
}
5

sure.

Mapping of the destination index:

{
  "product-definition-v19": {
    "mappings": {
      "dynamic": "false",
      "properties": {
        "_class": {
          "type": "keyword",
          "index": false,
          "doc_values": false
        },
        "aggregaterating": {
          "properties": {
            "bestrating": {
              "type": "float"
            },
            "itemreviewed": {
              "type": "text"
            },
            "ratingcount": {
              "type": "float"
            },
            "ratingvalue": {
              "type": "float"
            },
            "reviewcount": {
              "type": "float"
            },
            "type": {
              "type": "text"
            },
            "worstrating": {
              "type": "float"
            }
          }
        },
        "ai": {
          "properties": {
            "ClassifierResult": {
              "properties": {
                "cat1": {
                  "type": "keyword"
                },
                "cat2": {
                  "type": "keyword"
                },
                "cat3": {
                  "type": "keyword"
                },
                "cat4": {
                  "type": "keyword"
                },
                "cat5": {
                  "type": "keyword"
                },
                "cat6": {
                  "type": "keyword"
                },
                "cat7": {
                  "type": "keyword"
                },
                "cat8": {
                  "type": "keyword"
                }
              }
            },
            "NerResult": {
              "properties": {
                "brand": {
                  "type": "text"
                },
                "category": {
                  "type": "text"
                },
                "feature": {
                  "type": "text"
                },
                "model": {
                  "type": "text"
                },
                "prod_cat": {
                  "type": "text"
                },
                "product": {
                  "type": "text"
                }
              }
            },
            "PageQuality": {
              "type": "double"
            },
            "RefinedName": {
              "type": "text",
              "fields": {
                "1gram": {
                  "type": "text",
                  "analyzer": "first_1gram",
                  "search_analyzer": "first_1gram_search"
                },
                "2gram": {
                  "type": "text",
                  "analyzer": "first_2gram",
                  "search_analyzer": "first_2gram_search"
                },
                "3gram": {
                  "type": "text",
                  "analyzer": "first_3gram",
                  "search_analyzer": "first_3gram_search"
                }
              },
              "analyzer": "refined_name_analyzer",
              "search_analyzer": "refined_name_search_analyzer"
            },
            "RefinedTitle": {
              "type": "text",
              "analyzer": "refined_name_analyzer",
              "search_analyzer": "refined_name_search_analyzer"
            },
            "product": {
              "properties": {
                "popularity": {
                  "type": "double"
                },
                "score": {
                  "type": "double"
                }
              }
            },
            "version": {
              "type": "keyword"
            }
          }
        },
        "aiDesc": {
          "type": "text"
        },
        "altBrand": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          },
          "analyzer": "persian"
        },
        "autoSeoEnabled": {
          "type": "boolean"
        },
        "available": {
          "type": "boolean"
        },
        "availableShopCount": {
          "type": "integer"
        },
        "brand": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          },
          "analyzer": "persian"
        },
        "brandIdentifier": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          }
        },
        "cat1": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cat2": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cat3": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cat4": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cat5": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cat6": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cat7": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "cat8": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "categoriesIdentifier": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          }
        },
        "classifier": {
          "properties": {
            "version": {
              "type": "keyword"
            }
          }
        },
        "creationTime": {
          "type": "date"
        },
        "discountModificationTime": {
          "type": "date"
        },
        "discountPercent": {
          "type": "byte"
        },
        "featured": {
          "type": "boolean"
        },
        "featuredLevel": {
          "type": "integer"
        },
        "image": {
          "type": "keyword",
          "ignore_above": 512
        },
        "metaTags": {
          "properties": {
            "collection": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "model": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          },
          "analyzer": "persian"
        },
        "modificationTime": {
          "type": "date"
        },
        "names": {
          "type": "text",
          "fields": {
            "1gram": {
              "type": "text",
              "analyzer": "first_1gram",
              "search_analyzer": "first_1gram_search"
            },
            "2gram": {
              "type": "text",
              "analyzer": "first_2gram",
              "search_analyzer": "first_2gram_search"
            },
            "3gram": {
              "type": "text",
              "analyzer": "first_3gram",
              "search_analyzer": "first_3gram_search"
            }
          },
          "analyzer": "name_analyzer",
          "search_analyzer": "name_search_analyzer"
        },
        "ner": {
          "properties": {
            "brand": {
              "type": "text"
            },
            "category": {
              "type": "text"
            },
            "feature": {
              "type": "text"
            },
            "model": {
              "type": "text"
            },
            "prod_cat": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword"
                }
              }
            },
            "product": {
              "type": "text"
            },
            "version": {
              "type": "keyword"
            }
          }
        },
        "nonRefinedBrand": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          },
          "analyzer": "persian"
        },
        "page-quality": {
          "type": "float"
        },
        "persist": {
          "type": "boolean"
        },
        "price": {
          "type": "long"
        },
        "properties": {
          "type": "text",
          "analyzer": "persian"
        },
        "providerUrl": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "rank": {
          "type": "rank_feature"
        },
        "rankableShopCount": {
          "type": "rank_feature"
        },
        "refinedBrand": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          },
          "analyzer": "persian"
        },
        "relatedHosts": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          },
          "analyzer": "persian"
        },
        "relatedNames": {
          "type": "text",
          "analyzer": "persian"
        },
        "relatedShopsIdentifier": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          }
        },
        "representativeShopIdentifier": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          }
        },
        "representativeShopName": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "seoEnabled": {
          "type": "boolean"
        },
        "tag": {
          "properties": {
            "version": {
              "type": "keyword"
            }
          }
        },
        "tags": {
          "properties": {
            "confidence": {
              "type": "double"
            },
            "id": {
              "type": "keyword"
            }
          }
        },
        "title": {
          "type": "text",
          "analyzer": "persian"
        },
        "torobId": {
          "type": "keyword"
        },
        "totalShopCount": {
          "type": "integer"
        },
        "url": {
          "type": "keyword",
          "ignore_above": 512
        },
        "visits": {
          "type": "integer"
        }
      }
    }
  }
}

A sample from source index

{
  "took" : 6,
  "timed_out" : false,
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "product-definition-v19",
        "_type" : "_doc",
        "_id" : "e40ee20e-6282-38f3-9933-3e750c6f3f69",
        "_score" : 1.0,
        "_source" : {
          "available" : true,
          "videos" : [ ],
          "representativeShopName" : "site.com",
          "cat8" : "none",
          "relatedHosts" : [
            "site.com"
          ],
          "cat6" : "none",
          "cat7" : "none",
          "aiTags" : [
            "dummy",
            "dummy",
            "dummy"
          ],
          "price" : 1111,
          "modificationTime" : "2024-01-08T13:22:01.491Z",
          "rank" : 0.5162081334432932,
          "model" : "dummy",
          "ner" : {
            "product" : "dummy",
            "feature" : "dummy",
            "model" : "dummy",
            "store" : "",
            "none" : "dummy",
            "category" : "microusb",
            "brand" : "dummy",
            "prod_cat" : "dummy"
          },
          "id" : "e40ee20e-6282-38f3-9933-3e750c6f3f69",
          "brand" : "dummy",
          "categoriesIdentifier" : [
            9390,
            4161
          ],
          "image" : [
          ],
          "images" : [
          ],
          "ner.version" : "v2.14.1.3",
          "totalShopCount" : 1,
          "discountModificationTime" : "2023-10-07T09:09:39.166Z",
          "relatedShopsIdentifier" : [
            "2"
          ],
          "providerUrl" : [
            "site.com"
          ],
          "tags" : [
            {
              "confidence" : 0.932,
              "id" : "e40ee20e-6282-38f3-9933-3e750c6f3f69"
            }
          ],
          "cat4" : "none",
          "cat5" : "none",
          "cat2" : "none",
          "cat3" : "none",
          "cat1" : "none",
          "classifier.version" : "v2.18.5.1",

          "creationTime" : "2023-10-07T09:09:39.166Z",
          "description" : "description",
          "title" : [
            "title"
          ],
          "visits" : 0,
          "rankableShopCount" : 1,
          "discountPercent" : 0,
          "relatedNames" : [
            "a name"
          ],
          "confidence" : 0,
          "tag.version" : "v2.10.3",
          "url" : "",
          "names" : [
            "a name "
          ],
          "accessory_check" : "0",
          "availableShopCount" : 1,
          "attributes" : { },
          "aggregateRating" : {
            "reviewcount" : 1.0,
            "ratingvalue" : 4.0,
            "ratingcount" : 0.0,
            "worstrating" : 0.0,
            "type" : "AggregateRating",
            "bestrating" : 5.0
          },
          "brandIdentifier" : "15513",
          "properties" : [ ]
        }
      }
    ]
  }
}

I have to exclude some fields
because destination is empty is currently I include a doc from source

6

Yeah, sorry just got the error right.

Just tested it here on how to get a similar error, and the issue seems to be in the source document that has a different mapping as the expected in the destination.

This message:

object mapping for [tags] tried to parse field [null] as object, but found a concrete value

Really means that the destination mapping is an object, but the in the source document that logstash is trying to index the same field is not an object, but a keyword, something is not right.

Normally in the same log line you will have more information, like how the document looks like, can you look in the logs and share the entire log error for this issue?

Alternatively you can add a file our stdout output to see what logstash is really sending in the output.

As mentioned the tags field is an internal field used by logstash and using it may lead to issues.

Since you are migrating from 7 to 8, did you disabled the ecs_compatibility globally in logstash.yml?

7

@leandrojmp Thank you for putting your time and effort into this.

I disabled ecs_compatibility only in input and output filter

Here is an error from logstash I only distort the values

[2024-02-28T10:56:20,126][WARN ][logstash.outputs.elasticsearch][main][f367304aa46f25a8a19b6e9cdf76f288f96647dc34981825e117e9dbb0091be8] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"2c4ccd16-4304-3dda-a0bd-da00b470921c", :_index=>"product-definition-v19", :routing=>nil}, {"cat6"=>"none", "id"=>"2c4ccd16-4304-3dda-a0bd-da00b470921c", "image"=>"site.com", "cat4"=>"dummy", "aggregaterating"=>nil, "tag.version"=>"v2.10.3", "categoriesIdentifier"=>[2825, 3345, 3398, 3417, 3420], "_class"=>"ir.sobhan.sobhan_product_definition_handler.model.elastic.ProductDefinition", "@version"=>"1", "representativeShopName"=>"site.com", "cat7"=>"none", "_tags"=>[[{"confidence"=>0.932e0, "id"=>"2c4ccd16-4304-3dda-a0bd-da00b470921c"}]], "creationTime"=>"2022-11-09T19:16:32.132Z", "tags"=>["_tagsparsefailure"], "seoEnabled"=>false, "cat1"=>"dummy", "ai"=>{"RefinedTitle"=>"dummy", "NameAnalyze"=>{"SuggestedBrands"=>[], "product_coverted"=>false, "ProductExists"=>true, "BrandExists"=>false, "RefinedBrand"=>"", "pbm_standard"=>"dummy", "pb_standard"=>"dummy", "related_prodcats"=>[], "ModelExists"=>false, "related_digikala_prodcat"=>"", "score"=>2}, "product"=>{"score"=>0.104e0, "url_score"=>0, "popularity"=>0}, "PageQuality"=>1, "RefinedNameB"=>"dummy", "ClassifierResult"=>{"confidence"=>1, "cat7"=>"none", "cat3"=>"dummy", "cat4"=>"dummy", "accessory_check"=>"0", "cat6"=>"none", "cat8"=>"none", "cat5"=>"dummy", "cat1"=>"dummy", "cat2"=>"dummy"}, "RefinedTitleB"=>"dummy", "RefinedName"=>"dummy", "NerResult"=>{"model"=>"", "feature"=>"dummy", "category"=>"", "prod_cat"=>"dummy", "product"=>"dummy", "brand"=>"", "none"=>"کد"}, "ProductRank"=>3, "BookRefined"=>false}, "brandIdentifier"=>"15513", "available"=>false, "cat3"=>"dummy", "relatedShopsIdentifier"=>["1"], "cat5"=>"dummy", "breadcrumblist"=>[{"title"=>"dummy", "url"=>{"uri"=>"/", "base"=>nil}}, {"title"=>" dummy", "url"=>{"uri"=>"/main/apparel/", "base"=>nil}}, {"title"=>"dummy", "url"=>{"uri"=>"/url/path", "base"=>nil}}, {"title"=>"dummy", "url"=>{"uri"=>"/url/", "base"=>nil}}, {"title"=>"dummy", "url"=>{"uri"=>"/url/", "base"=>nil}}, {"title"=>"dummy", "url"=>{"uri"=>"/url/", "base"=>nil}}, {"title"=>"dummy", "url"=>{"uri"=>"/url/path", "base"=>nil}}], "relatedHosts"=>["site.com"], "brandReviewed"=>false, "persist"=>false, "featured"=>false, "url"=>"", "model"=>"s00001", "ner.version"=>"v2.14.1.3", "providerUrl"=>["site.com"], "attributes"=>{}, "images"=>["site.com"], "totalShopCount"=>1, "classifier.version"=>"v2.18.5.1", "featuredLevel"=>0, "rank"=>0.21242115649715045e1, "videos"=>[], "refinedCategory"=>"dummy", "ner"=>{"model"=>"", "feature"=>"dummy", "category"=>"", "prod_cat"=>"dummy", "product"=>"dummy", "store"=>"", "brand"=>"", "none"=>"dummy"}, "cat2"=>"dummy", "description"=>"dummy", "altBrand"=>"dummy", "accessory_check"=>"0", "title"=>nil, "properties"=>["dummy"], "brand"=>"dummy", "names"=>["dummy"], "subCategory"=>"dummy", "representativeShopIdentifier"=>"1", "representativeShopNameFa"=>"dummy", "confidence"=>1, "availableShopCount"=>0, "relatedNames"=>["dummy"], "cat8"=>"none", "modificationTime"=>"2024-01-08T06:45:12.723Z", "@timestamp"=>2024-02-28T07:26:19.945750720Z, "aiTags"=>["dummy"], "favoritePrice"=>0, "ai.version"=>"v2.14.4.6"}], :response=>{"index"=>{"_index"=>"product-definition-v19", "_id"=>"2c4ccd16-4304-3dda-a0bd-da00b470921c", "status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:3448] object mapping for [tags] tried to parse field [null] as object, but found a concrete value"}}}}
8

Yeah, in your document the tags field is not an object.

Here is its value:

"tags"=>["_tagsparsefailure"]

You have another similar field named _tags

"_tags"=>[[{"confidence"=>0.932e0, "id"=>"2c4ccd16-4304-3dda-a0bd-da00b470921c"}]]

Not sure what is causing it, but I would say that your original tags field is conflicting with the logstash internal tags field.

As mentioned the tags field is a base ecs field that should be a keyword field and it is used like this in all tools in the stack.

I'm also not sure if it is possible to solve this using Logstash, you may try adding the following filter to see if it works:

filter {
    mutate {
        rename => { "_tags" => "tags" }
    }
}

If it does not work I think you will need to whitelist the source IP and use reindex from remote or use a snapshot.

1 Like
9

Thank you so much I appreciate your help.
It is odd in the source index we don't have a field named _tags I think Logstash added that to the event.
I'll try your suggestions and let you know.

10

@leandrojmp unfortunately it didn't work. But I guess your are right its because of tags that makes Logstash confused.
I use elasticdump for migration, it is very slow but it works.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.