Reindexing 1.7 to 2.4.4 fails

Marjana · July 26, 2017, 8:25pm

Hi,
I am using migration plugin to reindex 1.7.1 indices to 2.4.4, so we can migrate to 5.x.
I managed to reindex some indices but most of them are failing. New index gets created but reindex fails. I tried to do it manually, copy>pasting the reindex command from the migration tool (and changing wait_for_completion to true).
Here is an example of one command and the output I get:

curl -XPOST 'http://elasearch03001.pod01:9201/_reindex?wait_for_completion=true' -d '
{
"source": {
"index": "sessions_org4_2017_07_17",
"sort": "_doc",
"size": 1000
},
"dest": {
"index": "sessions_org4_2017_07_17-2.4.4",
"version_type": "external"
}
}
'

What does this output mean "source is missing"? Both indices are present. One 2.4.4 has 0 docs, while the original has 57794 docs.

{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: source is missing;2: source is missing;3: source is missing;4: source is missing;5: source is missing;6: source is missing;7: source is missing;8: source is missing;9: source is missing;10: source is missing;11: source is missing;12: source is missing;13: source is missing;14:.....source is missing;724: source is missing;725: source is missing;726: source is missing;727: source is missing;"},"status":400}

Thanks

warkolm · July 26, 2017, 9:41pm

What does the mapping for the source index look like?

Marjana · July 26, 2017, 9:53pm

"sessions_org4_2017_07_17":{"mappings":{"_default_":{"_all":{"enabled":false},"_timestamp":{},"dynamic_templates":[{"event":{"mapping":{"index":"not_analyzed","store":false,"type":"integer"},"path_match":"Events.*"}},{"session_attribute":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"path_match":"SessionAttributes.*"}},{"url_argument":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"path_match":"screenViews.args.*"}},{"http_header":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"path_match":"httpHeaders.*"}},{"cookie":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"path_match":"cookies.*"}},{"dimension":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"match":"Dim*"}},{"generic":{"mapping":{"index":"no","store":false,"type":"{dynamic_type}"},"match":"@&~((Events|SessionAttributes)\\.*)","match_pattern":"regex"}}]},"freetexthit":{"_all":{"enabled":false},"_parent":{"type":"hit"},"_routing":{"required":true},"_source":{"enabled":false},"_timestamp":{},"dynamic_templates":[{"event":{"mapping":{"index":"not_analyzed","store":false,"type":"integer"},"path_match":"Events.*"}},{"session_attribute":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"path_match":"SessionAttributes.*"}},{"url_argument":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"path_match":"screenViews.args.*"}},{"http_header":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"path_match":"httpHeaders.*"}},{"cookie":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"path_match":"cookies.*"}},{"dimension":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"match":"Dim*"}},{"generic":{"mapping":{"index":"no","store":false,"type":"{dynamic_type}"},"match":"@&~((Events|SessionAttributes)\\.*)","match_pattern":"regex"}}],"properties":{"OrgCode":{"type":"string","analyzer":"custom_lowercase"},"freeText":{"type":"string","analyzer":"standard"}}},"hit":{"_all":{"enabled":false},"_parent":{"type":"session"},"_routing":{"required":true},"_timestamp":{},"dynamic_templates":[{"event":{"mapping":{"index":"not_analyzed","store":false,"type":"integer"},"path_match":"Events.*"}},{"session_attribute":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"path_match":"SessionAttributes.*"}},{"url_argument":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"path_match":"screenViews.args.*"}},{"http_header":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"path_match":"httpHeaders.*"}},{"cookie":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"path_match":"cookies.*"}},{"dimension":{"mapping":{"index":"analyzed","store":false,"analyzer":"custom_lowercase","type":"string"},"match":"Dim*"}},{"generic":{"mapping":{"index":"no","store":false,"type":"{dynamic_type}"},"match":"@&~((Events|SessionAttributes)\\.*)","match_pattern":"regex"}}],"properties":{"CollectorDCName":{"type":"string","index":"no"},"CollectorNodeID":{"type":"string","index":"no"},"Cookies":{"type":"nested","properties":{"Name":{"type":"string","index":"not_analyzed"},"Value":{"type":"string","analyzer":"custom_lowercase"}}},"HitDataType":{"type":"string","index":"no"},"HitNumber":{"type":"long"},"HitSource":{"type":"string","index":"no"},"HitTime":{"type":"long"},"HttpHeaders":{"type":"nested","properties":{"Name":{"type":"string","index":"not_analyzed"},"Value":{"type":"string","analyzer":"custom_lowercase"}}},"KafkaOffset":{"type":"long"},"KafkaPartition":{"type":"integer"},"KafkaTopic":{"type":"string","index":"not_analyzed"},"OrgCode":{"type":"string","analyzer":"custom_lowercase"},"ProcID":{"type":"integer"},"ProcServerID":{"type":"integer"},"ScreenViewCount":{"type":"integer"},"ScreenViewSources":{"properties":{"Count":{"type":"integer"},"Name":{"type":"string","index":"no"}}},"TLTHID":{"type":"string","index":"not_analyzed"},"TLTSID":{"type":"string","analyzer":"custom_lowercase"},"TimeBlock":{"properties":{"DAY_OF_MONTH":{"type":"byte"},"DAY_OF_WEEK":{"type":"byte"},"DAY_OF_YEAR":{"type":"short"},"HOUR_OF_DAY":{"type":"byte"},"MONTH":{"type":"byte"},"QUARTER":{"type":"byte"},"WEEK_OF_YEAR":{"type":"byte"},"YEAR":{"type":"short"}}},"facts":{"type":"nested","properties":{"Dimensions":{"type":"nested","properties":{"ID":{"type":"integer"},"Value":{"type":"string","analyzer":"custom_lowercase"}}},"EventID":{"type":"long"},"NumericValue":{"type":"double"},"StepNumber":{"type":"long"},"Value":{"type":"string","analyzer":"custom_lowercase"}}},"freeText":{"type":"string","analyzer":"standard"},"screenViews":{"properties":{"arguments":{"type":"nested","properties":{"Name":{"type":"string","index":"not_analyzed"},"Value":{"type":"string","analyzer":"custom_lowercase"}}},"fragment":{"type":"string","analyzer":"custom_lowercase"},"name":{"type":"string","analyzer":"custom_url"},"queryString":{"type":"string"}}}}},"session":{"_all":{"enabled":false},"_timestamp":{},"dynamic_templates":[{"event":{"mapping":{"index":"not_analyzed","store":false,"type":"integer"},"path_match":"Events.*"}},

Marjana · July 26, 2017, 9:53pm

Marjana · July 26, 2017, 9:54pm

too long to place it in one post, had to break it in two.

warkolm · July 26, 2017, 11:05pm

Please format it with code tags

Marjana · July 27, 2017, 10:29am

Sorry about that, it looks better now.

Christian_Dahlqvist · July 27, 2017, 10:33am

It is still hard to read, but there seems to be something in your index that has _source disabled, which will prevent it from being reindexed.

Marjana · July 27, 2017, 10:44am

Yes, I see it too now, freetexthit field.
Thank you Christian.

system · August 24, 2017, 10:44am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Upgrade from 1.7.2 to 5.4.0 Elasticsearch	3	637	July 21, 2017
Error while reindexing from 1.7 to 2.4 Elasticsearch	2	844	December 27, 2018
Debugging Partially Complete Reindexing Task? Elasticsearch	2	776	February 6, 2019
Reindex api request failing on Elasticsearch 6.6 Elasticsearch	5	773	December 16, 2021
_reindex API issue Elasticsearch	4	759	July 5, 2017

Reindexing 1.7 to 2.4.4 fails

Related topics