Elasticsearch Reindex Big Index

[eng.behjati (Nahid)]

Hi everyone!

I have a problem with elasticsearch reindex API.
when I run reindex API on big index , after ES reindexes certain number of documents, it responses "search context not found" exception. I should mention that I use python client.
I know this arise because scroll time expire but I could not find any way to set scroll in reindex API parameters to avoid this.
what should I do?

#reindex

[jpountz (Adrien Grand)]

Even on a large index, this should not occur. What does the reindex call look like?

[worroc]

Hi All,
Have the same issue. I reindex index with 23033921 documents but new index contains only 10925105.
I have run ~15 tasks 10 slices each.
in total it should reindex 222701787 documents but only 153689150 were actualy copied.
69,012,637 is missing.

How can we fix it?

Here is a task log,

samples
{
"slice_id": 5,
"total": 1896711,
"updated": 0,
"created": 30000,
"deleted": 0,
"batches": 3,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1,
"throttled_until_millis": 0
},

error
{
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": -1,
"index": null,
"reason": {
"type": "search_context_missing_exception",
"reason": "No search context found for id [69020]"
}
},

"error": {
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": -1,
"index": null,
"reason": {
"type": "search_context_missing_exception",
"reason": "No search context found for id [69021]"
}
},

#reindex

[worroc]

the elastic version is 5.6.3

[eng.behjati (Nahid)]

Hi jpountz!
thanks for your attention.

I used python client.
first I used directly reindex api and that was look like this:
elastic_client.reindex(
{
"source": {
"index": old_index_name
},
"dest": {
"index": new_index_name
}
})
it didn't work. so I changed it to this and it's looks like this :

helpers.reindex(elastic_client,
source_index=old_index_name,
target_index=new_index_name, query={"query": {
"match_all": {} }},
chunk_size = 1000,
scroll = "1h")

but still I get that error that says "No search context found for id [69021]" after almost half of index reindexed.

also my elasticsearch version is 5.5.1