# Reindexed Documents are not showing up, Response says it has created but it does not show up in destination index

**URL:** <https://discuss.elastic.co/t/reindexed-documents-are-not-showing-up-response-says-it-has-created-but-it-does-not-show-up-in-destination-index/337987>\
**Category:** Elasticsearch\
**Created:** [July 10, 2023, 5:50am UTC](https://discuss.elastic.co/t/reindexed-documents-are-not-showing-up-response-says-it-has-created-but-it-does-not-show-up-in-destination-index/337987 "2023-07-10T05:50:38Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aditya\_Teltia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aditya_teltia/32/122447_2.png) [@Aditya\_Teltia](https://discuss.elastic.co/u/Aditya_Teltia)\
**Post date:** [July 10, 2023, 5:50am UTC](https://discuss.elastic.co/t/reindexed-documents-are-not-showing-up-response-says-it-has-created-but-it-does-not-show-up-in-destination-index/337987/1 "2023-07-10T05:50:38Z")

</div>

`http://localhost:9201/restored_index/_search`

**Response:**

```auto
{
    "took": 2,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "restored_index",
                "_type": "_doc",
                "_id": "11JUPokBSZ0LgEWBxERF",
                "_score": 1.0,
                "_source": {
                    "foo": "bar"
                }
            }
        ]
    }
}

```

`http://localhost:9200/my-idx-new-09-2023/_search`

**Response:**

```auto
{
    "took": 6,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 10,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "2VJVPokBSZ0LgEWBM0Qi",
                "_score": 1.0,
                "_source": {
                    "name": "42a8edogab",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "2lJVPokBSZ0LgEWBN0Q8",
                "_score": 1.0,
                "_source": {
                    "name": "w8fmegu0j4",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "21JVPokBSZ0LgEWBO0RE",
                "_score": 1.0,
                "_source": {
                    "name": "8flxaze4f6",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "3FJVPokBSZ0LgEWBP0RN",
                "_score": 1.0,
                "_source": {
                    "name": "9g59tbgc1p",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "3VJVPokBSZ0LgEWBQ0RR",
                "_score": 1.0,
                "_source": {
                    "name": "nbicektp48",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "3lJVPokBSZ0LgEWBR0RQ",
                "_score": 1.0,
                "_source": {
                    "name": "42ry40id4e",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "31JVPokBSZ0LgEWBS0Ry",
                "_score": 1.0,
                "_source": {
                    "name": "atcwvil0hl",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "4FJVPokBSZ0LgEWBT0Rw",
                "_score": 1.0,
                "_source": {
                    "name": "cphxxk15lr",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "4VJVPokBSZ0LgEWBU0R8",
                "_score": 1.0,
                "_source": {
                    "name": "79yp7zrzwx",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            },
            {
                "_index": "my-idx-new-09-2023",
                "_type": "_doc",
                "_id": "4lJVPokBSZ0LgEWBV0SK",
                "_score": 1.0,
                "_source": {
                    "name": "olxrr29cz1",
                    "age": 30,
                    "email": "johndoe@example.com"
                }
            }
        ]
    }
}

```

```auto
http://localhost:9201/_reindex?wait_for_completion=true
{
    "source": {
        "remote": {
            "host": "http://localhost:9200"
        },
        "index": "my-idx-new-09-2023"
    },
    "dest": {
        "index": "restored_index"
    }
}

```

**Response:** [It is saying updated because it is already been created but it cannot be seen in `restored_index` why this is happening ?

```auto
{
    "took": 45,
    "timed_out": false,
    "total": 10,
    "updated": 10,
    "created": 0,
    "deleted": 0,
    "batches": 1,
    "version_conflicts": 0,
    "noops": 0,
    "retries": {
        "bulk": 0,
        "search": 0
    },
    "throttled_millis": 0,
    "requests_per_second": -1.0,
    "throttled_until_millis": 0,
    "failures": []
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 10, 2023, 5:59am UTC](https://discuss.elastic.co/t/reindexed-documents-are-not-showing-up-response-says-it-has-created-but-it-does-not-show-up-in-destination-index/337987/2 "2023-07-10T05:59:18Z")

</div>

> [@Aditya\_Teltia](#):
>
> [It is saying updated because it is already been created but it cannot be seen in `restored_index` why this is happening

What does "cannot be seen" mean

What command did you run?

Did you try to `_refresh` the index before you searched it?

---

<div class="post-metadata">

**Author:** ![Aditya\_Teltia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aditya_teltia/32/122447_2.png) [@Aditya\_Teltia](https://discuss.elastic.co/u/Aditya_Teltia)\
**Post date:** [July 10, 2023, 6:01am UTC](https://discuss.elastic.co/t/reindexed-documents-are-not-showing-up-response-says-it-has-created-but-it-does-not-show-up-in-destination-index/337987/3 "2023-07-10T06:01:28Z")

</div>

> [@stephenb](#):
>
> Did you try to `_refresh` the index

Yes I did. Apparently still the `restored_index` has only 1 document.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 10, 2023, 6:06am UTC](https://discuss.elastic.co/t/reindexed-documents-are-not-showing-up-response-says-it-has-created-but-it-does-not-show-up-in-destination-index/337987/4 "2023-07-10T06:06:12Z")

</div>

I'm Still confused about the 2 endpoints make it work on 1 ... Then try remote would be my suggestion ... And you fixed the pipeline???

Something else strange because it's says updated when it should say created assuming the docs are already not there

Is your default\_pipeline routing those docs elsewhere?

There is something fundamental wrong.

You can also set `op_type` to create to only allow new docs to be created to debug

> Setting op\_type to create causes \_reindex to only create missing documents in the destination. All existing documents will cause a version conflict.

---

<div class="post-metadata">

**Author:** ![Aditya\_Teltia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aditya_teltia/32/122447_2.png) [@Aditya\_Teltia](https://discuss.elastic.co/u/Aditya_Teltia)\
**Post date:** [July 10, 2023, 6:20am UTC](https://discuss.elastic.co/t/reindexed-documents-are-not-showing-up-response-says-it-has-created-but-it-does-not-show-up-in-destination-index/337987/5 "2023-07-10T06:20:20Z")

</div>

I think I have a \_ingest pipeline set

```auto
http://localhost:9201/_ingest/pipeline/write-redirect-pipeline
{
  "processors" : [
    {
      "set" : {
        "field": "_index",
        "value": "my-idx-new-09-2023"
      }
    }
  ]
}

```

So all the reindexed documents are going to `my-idx-new-09-2023` but I have a doubt that I have not associated this write-pipeline in the settings of my `restored_index `.

What I mean to say is didn't explicitly did this

```auto
http://localhost:9201/restored_index/_settings
{
   "index": {
      "default_pipeline": write-redirect-pipeline
    }
}

```

So why is it showing up in the settings of my `restored_index`. Is this the default behaviour ?

```auto
http://localhost:9201/restored_index/_settings

```

**Response:**

```auto
{
    "restored_index": {
        "settings": {
            "index": {
                "routing": {
                    "allocation": {
                        "include": {
                            "_tier_preference": "data_content"
                        }
                    }
                },
                "number_of_shards": "1",
                "provided_name": "my-idx-09-2022",
                "default_pipeline": "write_redirect_pipeline",
                "creation_date": "1688967889832",
                "number_of_replicas": "1",
                "uuid": "JUhpWka6SGeh01F384WSmA",
                "version": {
                    "created": "7171099"
                }
            }
        }
    }
}

```

Initally the error was `write-redirect-pipeline` does not exist which got resolved when I created it in my cluster2. But now it is getting associated with my `restored_index` even if I am not doing it explicitly and all the reindexed documents are indexing to `my-idx-new-09-2023`.

Can I do this some other way around ?

---

<div class="post-metadata">

**Author:** ![Aditya\_Teltia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aditya_teltia/32/122447_2.png) [@Aditya\_Teltia](https://discuss.elastic.co/u/Aditya_Teltia)\
**Post date:** [July 10, 2023, 6:24am UTC](https://discuss.elastic.co/t/reindexed-documents-are-not-showing-up-response-says-it-has-created-but-it-does-not-show-up-in-destination-index/337987/6 "2023-07-10T06:24:24Z")

</div>

I just want to reindex ` my-idx-new-09-2023` in cluster1 which has been created by ingest pipeline from index `my-idx-09-2022` (also in cluster1) to an index `restored_index` in cluster2.

Because this ` my-idx-new-09-2023` has been created by ingest pipeline. I have to create the same pipeline else if shows `pipeline does not exist`. But I am not setting the pipeline to `restored_index` and using reindex api.

I think, I cannot use reroute because I am using 7.17.10 version of Elasticsearch.

---

<div class="post-metadata">

**Author:** ![Aditya\_Teltia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aditya_teltia/32/122447_2.png) [@Aditya\_Teltia](https://discuss.elastic.co/u/Aditya_Teltia)\
**Post date:** [July 10, 2023, 6:49am UTC](https://discuss.elastic.co/t/reindexed-documents-are-not-showing-up-response-says-it-has-created-but-it-does-not-show-up-in-destination-index/337987/7 "2023-07-10T06:49:33Z")

</div>

Apologies for this irrelevant question. I was stuck with this bug for a long time and wasn't able to figure out what was happening. Turns out I was doing something fundamentally wrong, I have fixed it and it is working now.

Thank you @stephenb for the help and apologies for wasting your time.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 10, 2023, 1:37pm UTC](https://discuss.elastic.co/t/reindexed-documents-are-not-showing-up-response-says-it-has-created-but-it-does-not-show-up-in-destination-index/337987/8 "2023-07-10T13:37:39Z")

</div>

> [@Aditya\_Teltia](#):
>
> Initally the error was `write-redirect-pipeline` does not exist which got resolved when I created it in my cluster2. But now it is getting associated with my `restored_index` even if I am not doing it explicitly and all the reindexed documents are indexing to `my-idx-new-09-2023`.

That is because when you restore an index the index settings from the source index are restored as well.

In this case the `default_pipeline`

In the restore you could have set `ignore_index_settings` then that setting would not have been restored... Or you could have simply removed it afterward

> [@Aditya\_Teltia](#):
>
> I have to create the same pipeline else if shows `pipeline does not exist`. But I am not setting the pipeline to `restored_index` and using reindex api.

Correct. But since as we just discussed the default\_pipeline is already set for the restored index the pipeline will be executed anyways for every document coming into that index, whether it's directly into it or through a reindex

> [@Aditya\_Teltia](#):
>
> I think, I cannot use reroute because I am using 7.17.10 version of Elasticsearch.

Correct that is a new 8.x feature

> [@Aditya\_Teltia](#):
>
> Apologies for this irrelevant question. I was stuck with this bug for a long time and wasn't able to figure out what was happening. Turns out I was doing something fundamentally wrong, I have fixed it and it is working now.

No waste of time but we would ask you to tell us what the error was so others can learn from it, especially after such a complete discussion.

---

<div class="post-metadata">

**Author:** ![Aditya\_Teltia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aditya_teltia/32/122447_2.png) [@Aditya\_Teltia](https://discuss.elastic.co/u/Aditya_Teltia)\
**Post date:** [July 10, 2023, 2:09pm UTC](https://discuss.elastic.co/t/reindexed-documents-are-not-showing-up-response-says-it-has-created-but-it-does-not-show-up-in-destination-index/337987/9 "2023-07-10T14:09:28Z")

</div>

> [@stephenb](#):
>
> No waste of time but we would ask you to tell us what the error was so others can learn from it, especially after such a complete discussion.

Sure, it was a coding error, I was aware that when we restore an index its settings also get restored still I don't know why but I forgot it in this case that `default_pipeline` will also be restored to the destination index i.e `restored_index`. Due to which I was unable to understand that even after the response of `_reindex` is saying that documents are created, why aren't they visible in destination index `restored_index`. It would have been better if I had checked the settings of my indices before asking doubt here. Apologies for that.

> [@stephenb](#):
>
> That is because when you restore an index the index settings from the source index are restored as well.

You mentioned it as well.

How I fixed it ?  
Simply set the `default_pipeline: null` before reindexing.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 10, 2023, 2:15pm UTC](https://discuss.elastic.co/t/reindexed-documents-are-not-showing-up-response-says-it-has-created-but-it-does-not-show-up-in-destination-index/337987/10 "2023-07-10T14:15:13Z")

</div>

> [@Aditya\_Teltia](#):
>
> How I fixed it ?  
> Simply set the `default_pipeline: null` before reindexing

Yes that should work... this is one of the pitfalls of `default_pipeline` ... it "lurks" in the background and you don't see / ask for it to run.. but it does anyways.

Or as I mentioned you can not restore the index settings in the first place...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 7, 2023, 2:15pm UTC](https://discuss.elastic.co/t/reindexed-documents-are-not-showing-up-response-says-it-has-created-but-it-does-not-show-up-in-destination-index/337987/11 "2023-08-07T14:15:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.