Reindexed Documents are not showing up, Response says it has created but it does not show up in destination index

Elastic Stack Elasticsearch
1

http://localhost:9201/restored_index/_search

Response:

{
    "took": 2,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "restored_index",
                "_type": "_doc",
                "_id": "11JUPokBSZ0LgEWBxERF",
                "_score": 1.0,
                "_source": {
                    "foo": "bar"
                }
            }
        ]
    }
}

http://localhost:9200/my-idx-new-09-2023/_search

Response:

{
    "took": 6,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 10,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "2VJVPokBSZ0LgEWBM0Qi",
                "_score": 1.0,
                "_source": {
                    "name": "42a8edogab",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "2lJVPokBSZ0LgEWBN0Q8",
                "_score": 1.0,
                "_source": {
                    "name": "w8fmegu0j4",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "21JVPokBSZ0LgEWBO0RE",
                "_score": 1.0,
                "_source": {
                    "name": "8flxaze4f6",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "3FJVPokBSZ0LgEWBP0RN",
                "_score": 1.0,
                "_source": {
                    "name": "9g59tbgc1p",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "3VJVPokBSZ0LgEWBQ0RR",
                "_score": 1.0,
                "_source": {
                    "name": "nbicektp48",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "3lJVPokBSZ0LgEWBR0RQ",
                "_score": 1.0,
                "_source": {
                    "name": "42ry40id4e",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "31JVPokBSZ0LgEWBS0Ry",
                "_score": 1.0,
                "_source": {
                    "name": "atcwvil0hl",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "4FJVPokBSZ0LgEWBT0Rw",
                "_score": 1.0,
                "_source": {
                    "name": "cphxxk15lr",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "4VJVPokBSZ0LgEWBU0R8",
                "_score": 1.0,
                "_source": {
                    "name": "79yp7zrzwx",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "4lJVPokBSZ0LgEWBV0SK",
                "_score": 1.0,
                "_source": {
                    "name": "olxrr29cz1",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            }
        ]
    }
}

http://localhost:9201/_reindex?wait_for_completion=true
{
    "source": {
        "remote": {
            "host": "http://localhost:9200"
        },
        "index": "my-idx-new-09-2023"
    },
    "dest": {
        "index": "restored_index"
    }
}

Response: [It is saying updated because it is already been created but it cannot be seen in restored_index why this is happening ?

{
    "took": 45,
    "timed_out": false,
    "total": 10,
    "updated": 10,
    "created": 0,
    "deleted": 0,
    "batches": 1,
    "version_conflicts": 0,
    "noops": 0,
    "retries": {
        "bulk": 0,
        "search": 0
    },
    "throttled_millis": 0,
    "requests_per_second": -1.0,
    "throttled_until_millis": 0,
    "failures": []
}
2

What does "cannot be seen" mean

What command did you run?

Did you try to _refresh the index before you searched it?

3

Yes I did. Apparently still the restored_index has only 1 document.

4

I'm Still confused about the 2 endpoints make it work on 1 ... Then try remote would be my suggestion ... And you fixed the pipeline???

Something else strange because it's says updated when it should say created assuming the docs are already not there

Is your default_pipeline routing those docs elsewhere?

There is something fundamental wrong.

You can also set op_type to create to only allow new docs to be created to debug

Setting op_type to create causes _reindex to only create missing documents in the destination. All existing documents will cause a version conflict.

1 Like
5

I think I have a _ingest pipeline set

http://localhost:9201/_ingest/pipeline/write-redirect-pipeline
{
  "processors" : [
    {
      "set" : {
        "field": "_index",
        "value": "my-idx-new-09-2023"
      }
    }
  ]
}

So all the reindexed documents are going to my-idx-new-09-2023 but I have a doubt that I have not associated this write-pipeline in the settings of my restored_index .

What I mean to say is didn't explicitly did this

http://localhost:9201/restored_index/_settings
{
   "index": {
      "default_pipeline": write-redirect-pipeline
    }
}

So why is it showing up in the settings of my restored_index. Is this the default behaviour ?

http://localhost:9201/restored_index/_settings

Response:

{
    "restored_index": {
        "settings": {
            "index": {
                "routing": {
                    "allocation": {
                        "include": {
                            "_tier_preference": "data_content"
                        }
                    }
                },
                "number_of_shards": "1",
                "provided_name": "my-idx-09-2022",
                "default_pipeline": "write_redirect_pipeline",
                "creation_date": "1688967889832",
                "number_of_replicas": "1",
                "uuid": "JUhpWka6SGeh01F384WSmA",
                "version": {
                    "created": "7171099"
                }
            }
        }
    }
}

Initally the error was write-redirect-pipeline does not exist which got resolved when I created it in my cluster2. But now it is getting associated with my restored_index even if I am not doing it explicitly and all the reindexed documents are indexing to my-idx-new-09-2023.

Can I do this some other way around ?

6

I just want to reindex my-idx-new-09-2023 in cluster1 which has been created by ingest pipeline from index my-idx-09-2022 (also in cluster1) to an index restored_index in cluster2.

Because this my-idx-new-09-2023 has been created by ingest pipeline. I have to create the same pipeline else if shows pipeline does not exist. But I am not setting the pipeline to restored_index and using reindex api.

I think, I cannot use reroute because I am using 7.17.10 version of Elasticsearch.

7

Apologies for this irrelevant question. I was stuck with this bug for a long time and wasn't able to figure out what was happening. Turns out I was doing something fundamentally wrong, I have fixed it and it is working now.

Thank you @stephenb for the help and apologies for wasting your time.

1 Like
8

That is because when you restore an index the index settings from the source index are restored as well.

In this case the default_pipeline

In the restore you could have set ignore_index_settings then that setting would not have been restored... Or you could have simply removed it afterward

Correct. But since as we just discussed the default_pipeline is already set for the restored index the pipeline will be executed anyways for every document coming into that index, whether it's directly into it or through a reindex

Correct that is a new 8.x feature

No waste of time but we would ask you to tell us what the error was so others can learn from it, especially after such a complete discussion.

1 Like
9

Sure, it was a coding error, I was aware that when we restore an index its settings also get restored still I don't know why but I forgot it in this case that default_pipeline will also be restored to the destination index i.e restored_index. Due to which I was unable to understand that even after the response of _reindex is saying that documents are created, why aren't they visible in destination index restored_index. It would have been better if I had checked the settings of my indices before asking doubt here. Apologies for that.

You mentioned it as well.

How I fixed it ?
Simply set the default_pipeline: null before reindexing.

10

Yes that should work... this is one of the pitfalls of default_pipeline ... it "lurks" in the background and you don't see / ask for it to run.. but it does anyways.

Or as I mentioned you can not restore the index settings in the first place...

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.