# Losing documents while Reindex

**URL:** <https://discuss.elastic.co/t/losing-documents-while-reindex/158131>\
**Category:** Elasticsearch\
**Created:** [November 26, 2018, 8:15am UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131 "2018-11-26T08:15:10Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![smogami](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/smogami/32/26787_2.png) [@smogami](https://discuss.elastic.co/u/smogami)\
**Post date:** [November 26, 2018, 8:15am UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131/1 "2018-11-26T08:15:11Z")

</div>

- Elasticsearch: 6.2

Hello.

I am trying to update the mapping of an existing index, by the operations described below.  
This operation works well for the index if number of documents are relatively small (e.g. 1,000).  
However, if the index has many (e.g. 100,000) documents, 50～90% of documents are lost...

Any help?

* * *

Step 1. Update Index template

```auto
PUT _template/twitter -d @template_twitter.json

```

Step 2. Create new temporary index from existing index.

```auto
POST _reindex?wait_for_completion=true
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "twitter_temp"
  }
}

```

Step 3. Delete old index

```auto
DELETE twitter

```

Step 4. Restore from temporary index.

```auto
POST _reindex?wait_for_compretion=true
{
  "source": {
    "index": "twitter_temp"
  },
  "dest": {
    "index": "twitter"
  }
}

```

Step 5. Delete temporary index.

```auto
DELETE twitter_temp

```

* * *

After Step 2,

```auto
GET twitter/_stats

```

and

```auto
GET twitter_temp/_stats

```

returns JSON in which `_all.total.docs.count` field are different.

So I guess that I miss something in Step 2 so data losing happens.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 26, 2018, 8:23am UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131/2 "2018-11-26T08:23:56Z")

</div>

> [@smogami](#):
>
> wait\_for\_compretion

Should this not be `wait_for_completion`? Is this a copy and paste issue or what you have actually been running?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 26, 2018, 8:27am UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131/3 "2018-11-26T08:27:30Z")

</div>

Also could you run `_search?size=0` in both cases instead the index stats API?

---

<div class="post-metadata">

**Author:** ![smogami](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/smogami/32/26787_2.png) [@smogami](https://discuss.elastic.co/u/smogami)\
**Post date:** [November 27, 2018, 2:49am UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131/4 "2018-11-27T02:49:59Z")

</div>

> Should this not be `wait_for_completion` ?

This issue happens both if `wait_for_competion` exists or not.  
Actually, I added `wait_for_completion` hoping to resolve the issue, but the issue remains.

> Is this a copy and paste issue or what you have actually been running?  
> I don't understand what `copy and pase issue` means.

This is actual issue I am facing in my operation.  
As mentioned, this steps works for smaller index.  
The index name `twitter` is dummy, though.

---

<div class="post-metadata">

**Author:** ![smogami](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/smogami/32/26787_2.png) [@smogami](https://discuss.elastic.co/u/smogami)\
**Post date:** [November 27, 2018, 3:07am UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131/5 "2018-11-27T03:07:13Z")

</div>

Below is a result of `_search?size=0` for **original** index.

```auto
{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 24007,
    "max_score": 0,
    "hits": []
  }
}

```

Below is a result of `_search?size=0` for **temp** index.

```auto
{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 16294,
    "max_score": 0,
    "hits": []
  }
}

```

Count for original was 24007, but temp was 16294.

BTW, a result of `_reindex` (step 2) was below.  
It looks like 24007 created in temp successfully, but only 16294 returned for search.

```auto
{
  "took": 3656,
  "timed_out": false,
  "total": 24007,
  "updated": 0,
  "created": 24007,
  "deleted": 0,
  "batches": 25,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1,
  "throttled_until_millis": 0,
  "failures": []
}

```

---

<div class="post-metadata">

**Author:** ![johtani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johtani/32/44956_2.png) [@johtani](https://discuss.elastic.co/u/johtani)\
**Post date:** [November 27, 2018, 3:41am UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131/6 "2018-11-27T03:41:13Z")

</div>

Could you run below query to original index?

```auto
/_search
{
  "size": 0,
  "aggs":{
    "fuga": {
      "cardinality": {
        "field": "_id",
        "precision_threshold": 25000
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![smogami](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/smogami/32/26787_2.png) [@smogami](https://discuss.elastic.co/u/smogami)\
**Post date:** [November 27, 2018, 4:43am UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131/7 "2018-11-27T04:43:23Z")

</div>

Actually `"hits"` have more documents but omitted, since the forum has 7,000 characters limitation in one post.

```auto
{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 24007,
    "max_score": 1,
    "hits": [
      {
        "_index": "dev.annotation_v2",
        "_type": "annotation_type",
        "_id": "[\"410f580a-6183-4a4c-94fd-5083cfcd971b\",\"test_2\",\"dafbdf3d-dea6-45d9-8775-1ac89a603c0d\",\"cf031890-c155-4767-af3b-3b2c26b54a37\"]",
        "_score": 1,
        "_source": {
          "project_id": "410f580a-6183-4a4c-94fd-5083cfcd971b",
          "task_id": "test_2",
          "input_data_id": "dafbdf3d-dea6-45d9-8775-1ac89a603c0d",
          "detail": {
            "annotation_id": "cf031890-c155-4767-af3b-3b2c26b54a37",
            "account_id": "john_doe",
            "label_id": "a07c68fb-b8cd-499a-817f-c2bbb83007d2",
            "is_protected": false,
            "data_holding_type": "inner",
            "data": "70,103,165,138",
            "path": null,
            "etag": null,
            "additional_data_list": [
              {
                "additional_data_definition_id": "2d6b7295-6376-4887-ba1d-81bc43ac0de4",
                "flag": null,
                "integer": null,
                "comment": null,
                "choice": null
              }
            ],
            "created_datetime": "2018-11-12T13:32:14.061+09:00",
            "updated_datetime": "2018-11-12T13:32:14.061+09:00"
          },
          "updated_datetime": "2018-11-12T13:32:14.001+09:00",
          "@timestamp": "2018-11-12T04:32:18.300Z"
        }
      }
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 27, 2018, 6:57am UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131/8 "2018-11-27T06:57:34Z")

</div>

This id is weird

```
 "_id": "[\"410f580a-6183-4a4c-94fd-5083cfcd971b\",\"test_2\",\"dafbdf3d-dea6-45d9-8775-1ac89a603c0d\",\"cf031890-c155-4767-af3b-3b2c26b54a37\"]",

```

Could you check if it does exist in the original index?

---

<div class="post-metadata">

**Author:** ![smogami](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/smogami/32/26787_2.png) [@smogami](https://discuss.elastic.co/u/smogami)\
**Post date:** [November 27, 2018, 12:18pm UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131/9 "2018-11-27T12:18:07Z")

</div>

> [@dadoonet](#):
>
> "\_id": "["410f580a-6183-4a4c-94fd-5083cfcd971b","test\_2","dafbdf3d-dea6-45d9-8775-1ac89a603c0d","cf031890-c155-4767-af3b-3b2c26b54a37"]",

> Could you check if it does exist in the original index?

Both original and temp index has the document of this `_id`.

My app stores _compound_-key-ish value to `_id`, such as `["a","b","c","d"]`.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 27, 2018, 12:48pm UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131/10 "2018-11-27T12:48:52Z")

</div>

I'm wondering if this kind of `_id` could be a source of a bug somewhat...

@nik9000 does this ring a bell?

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [November 30, 2018, 3:16pm UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131/11 "2018-11-30T15:16:03Z")

</div>

I've never tried to use an id like this! It makes sense and I don't think it should be a problem.

This probably isn't it, but maybe add `refresh=true` to the `_reindex` just to make sure that the documents didn't make it. Reindex certainly thinks it wrote the right number of documents and I haven't seen it hit an issue with counting.

---

<div class="post-metadata">

**Author:** ![smogami](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/smogami/32/26787_2.png) [@smogami](https://discuss.elastic.co/u/smogami)\
**Post date:** [December 4, 2018, 9:00am UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131/12 "2018-12-04T09:00:05Z")

</div>

Sorry for late response.

Unfortunately, adding `?refresh` does not solve this issue.  
However, I deleted `twitter_temp` completely and run `_reindex`, then there were no document loss.  
(I kept the temp index to verify count matches with original index)

I assume my index are broken somehow...?

Anyway, I will close this issue.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 4, 2018, 9:12am UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131/13 "2018-12-04T09:12:31Z")

</div>

That can indicate that you had a wrong mapping (no mapping so using default). And while reindexing some documents failed the reindex operation because of that.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 1, 2019, 9:12am UTC](https://discuss.elastic.co/t/losing-documents-while-reindex/158131/14 "2019-01-01T09:12:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.