# Reindexing elastic search does not return all documents

**URL:** <https://discuss.elastic.co/t/reindexing-elastic-search-does-not-return-all-documents/246687>\
**Category:** Elasticsearch\
**Created:** [August 27, 2020, 9:13pm UTC](https://discuss.elastic.co/t/reindexing-elastic-search-does-not-return-all-documents/246687 "2020-08-27T21:13:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![byc-art](https://avatars.discourse-cdn.com/v4/letter/b/edb3f5/32.png) [@byc-art](https://discuss.elastic.co/u/byc-art)\
**Post date:** [August 27, 2020, 9:13pm UTC](https://discuss.elastic.co/t/reindexing-elastic-search-does-not-return-all-documents/246687/1 "2020-08-27T21:13:57Z")

</div>

0

I have about 1.5 million documents in my elastic search. I'm hoping to reindex them so that each index filters documents containing certain keywords, and one ( `null index` ) that do not contain any of the keywords I specified in other indices. I'm not sure why my indices returned fewer documents than expected. Particularly I'm expecting about 1.2 million documents in the `null index` but it only returned about 30k documents in the new index. Would appreciate ideas on what I've done wrong here!

This is how I reindex documents containing certain keywords in multiple fields

```auto
curl --location --request POST 'http://abcdef2344:9200/_reindex' \
--header 'Content-Type: application/json' \
--data-raw '{
  "source": {
    "index": "mydocs_email_*",
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "should": [
                {
                  "multi_match": {
                    "fields": [
                      "content",
                      "meta.raw.Message:Raw-Header:Subject"
                    ],
                    "query": "keyword1"
                  }
                },
                {
                  "multi_match": {
                    "fields": [
                      "content",
                      "meta.raw.Message:Raw-Header:Subject"
                    ],
                    "query": "keyword2"
                  }
                }
              ]
            }
          }
        ]
      }
    }
  },
  "dest": {
    "index": "analysis_keywords"
  }
}'

```

Then I use `must_not` to create another index that do not contain `keyword1` and `keyword2` .

```auto
curl --location --request POST 'http://abcdef2344:9200/_reindex' \
--header 'Content-Type: application/json' \
--data-raw '{
  "source": {
    "index": "mydocs_email_*",
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "must_not": [
                {
                  "multi_match": {
                    "fields": [
                      "content",
                      "meta.raw.Message:Raw-Header:Subject"
                    ],
                    "query": "keyword1"
                  }
                },
                {
                  "multi_match": {
                    "fields": [
                      "content",
                      "meta.raw.Message:Raw-Header:Subject"
                    ],
                    "query": "keyword2"
                  }
                }
              ]
            }
          }
        ]
      }
    }
  },
  "dest": {
    "index": "analysis_null"
  }
}'

```

The `null index` returned 29.7k documents. From the error message it looks like I should expect 1.28 million files. It also said I need to increase the number of fields in the index - which I also did after running the codes above. Though the number of files still stay the same.

```auto
{"took":53251,"timed_out":false,"total":1277428,"updated":243,"created":29755,"deleted":0,"batches":30,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[{"index":"analysis_null","type":"_doc","id":"/email/.......msg","cause":{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [analysis_null] has been exceeded"},"status":400}]

```

---

<div class="post-metadata">

**Author:** ![byc-art](https://avatars.discourse-cdn.com/v4/letter/b/edb3f5/32.png) [@byc-art](https://discuss.elastic.co/u/byc-art)\
**Post date:** [September 2, 2020, 9:44am UTC](https://discuss.elastic.co/t/reindexing-elastic-search-does-not-return-all-documents/246687/2 "2020-09-02T09:44:01Z")

</div>

I increased the field limit before reindexing and solved the issue

```auto
> DELETE analysis_null
> 
> PUT analysis_null
> {
> "settings": {
> "index.mapping.total_fields.limit": 10000
> }
> }

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 30, 2020, 9:44am UTC](https://discuss.elastic.co/t/reindexing-elastic-search-does-not-return-all-documents/246687/3 "2020-09-30T09:44:02Z")

</div>

Closed on September 30, 2020, 9:44am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.