Reindexing failed - Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 672;

Elastic Stack Elasticsearch
1

I am reindexing the elasticsearch with the cluster restart from 2.4.1 to 5.6.16, I have reindexed few indices successfull. Out of 23 indices 3 are getting failed duw to the Validation Failure with the id.

Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 672;

Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 549;2: id is too long, must be no longer than 512 bytes but was: 517;3: id is too long, must be no longer than 512bytes but was: 532;4: id is too long, must be no longer than 512 bytes but was: 719;5: id is too long, must be no longer than 512 bytes but was: 1374;6: id is too long, must be no longer than 512 bytes but was:614;7: id is too long, must be no longer than 512 bytes but was: 623;8: id is too long, must be no longer than 512 bytes but was: 543;9: id is too long, must be no longer than 512 bytes but was: 895;10: id is too long, must be no longer than 512 bytes but was: 697;11: id is too long, must be no longer than 512 bytes but was: 1064;12: id is too long, must be no longer than 512 bytes but was: 785;13: id is too long, must be no longer than 512 bytes but was: 600;14: id is too long, must be no longer than 512 bytes but was: 537;15: id is too long, must be no longer than 512 bytes but was: 870;16: id is too long, must be no longer than 512 bytes but was: 802;17: id is too long, must be no longer than 512 bytes but was: 716;18: id is too long, must be no longer than 512 bytes but was: 844;19: id is too long, must be no longer than 512 bytes but was: 1503;20: id is too long, must be no longer than 512 bytes but was: 734;21: id is too long, must be no longer than 512 bytes but was: 590;22: id is too long, must be no longer than 512 bytes but was:697;23: id is too long, must be no longer than 512 bytes but was: 1008;24: id is too long, must be no longer than 512 bytes but was: 1146;25: id is too long, must be no longer than 512 bytes but was: 609;26: idis too long, must be no longer than 512 bytes but was: 988;27: id is too long, must be no longer than 512 bytes but was: 703;28: id is too long, must be no longer than 512 bytes but was: 568;29: id is too long,must be no longer than 512 bytes but was: 726;30: id is too long, must be no longer than 512 bytes but was: 737;31: id is too long, must be no longer than 512 bytes but was: 611;32: id is too long, must be no longer than 512 bytes but was: 914;

2

This limit is hard-coded and can't be changed, so you'll need to find a way to reduce the length of the ID field.
A similar issue came up recently and it was noted that the limit isn't documented anywhere. I think the docs are in the process of being updated.

1 Like
3

Doing it manually is very infinite steps to identify the records, i have same data in other database along with elasticsearch.

Elasticsearch team may need to suggest something.

4

I solved this issue with replacing the _id value with new hash.

Issue Logs : Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 20456;2: id is too long, must be no longer than 512 bytes but was: 537;3: id is too long, must be no longer than 512 bytes but was: 10978;4: id is too long, must be no longer than 512 bytes but was: 544;

Solution :

  • Run the following reindexing script for the indexes whose _id is more than 512 bytes
  1. Index new_student

curl -X DELETE "localhost:9200/new_student?pretty=true"

curl -X POST "localhost:9200/_reindex?wait_for_completion=true&pretty=true" -H 'Content-Type: application/json' -d'
{
"source": {
"index": "student"
},
"dest": {
"index": "new_student"
},
"script": {
"source": "if (ctx._id.length() > 512) {ctx._id = 'null'}",
"lang": "painless"
}
}' >> student.txt

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.