# Number of documents in Dev tools is different from number of document in Discover

**URL:** <https://discuss.elastic.co/t/number-of-documents-in-dev-tools-is-different-from-number-of-document-in-discover/274917>\
**Category:** Kibana\
**Created:** [June 3, 2021, 7:55pm UTC](https://discuss.elastic.co/t/number-of-documents-in-dev-tools-is-different-from-number-of-document-in-discover/274917 "2021-06-03T19:55:36Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\
**Post date:** [June 3, 2021, 7:55pm UTC](https://discuss.elastic.co/t/number-of-documents-in-dev-tools-is-different-from-number-of-document-in-discover/274917/1 "2021-06-03T19:55:36Z")

</div>

I am using ELK 7.9.3 and I have an index which contain more then 2M documents, I used `reindex` API to have a sample using the following command:

```auto
POST /_reindex?wait_for_completion=false
{
  "max_docs": 1000, 
  "source": {
    "index": "firstIndex-2020.10.27-000313",
    "query": {
      "match_all": {}
    }
  },
  "dest": {
    "index": "develop1000"
  }
}

```

I am expected to have 1000 documents in the index named `develop1000`, that's the result of `GET develop1000/_count` but the problem is when I check in Discover, I only have 992 hits.

What could be the problem please? and how to investigate the issue? I have the same problem in another index!

Thanks folks.

---

<div class="post-metadata">

**Author:** ![AClerk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aclerk/32/55297_2.png) [@AClerk](https://discuss.elastic.co/u/AClerk)\
**Post date:** [June 4, 2021, 2:42am UTC](https://discuss.elastic.co/t/number-of-documents-in-dev-tools-is-different-from-number-of-document-in-discover/274917/2 "2021-06-04T02:42:57Z")

</div>

> [@marone](#):
>
> `firstIndex-2020.10.27-000313`

Can you show the number of docs? ⬆

---

<div class="post-metadata">

**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\
**Post date:** [June 4, 2021, 1:19pm UTC](https://discuss.elastic.co/t/number-of-documents-in-dev-tools-is-different-from-number-of-document-in-discover/274917/3 "2021-06-04T13:19:38Z")

</div>

Sure, below the total number of docs:

```auto
{
  "count" : 2652481,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  }
}

```

---

<div class="post-metadata">

**Author:** ![tmp13](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tmp13/32/78005_2.png) [@tmp13](https://discuss.elastic.co/u/tmp13)\
**Post date:** [June 6, 2021, 6:13pm UTC](https://discuss.elastic.co/t/number-of-documents-in-dev-tools-is-different-from-number-of-document-in-discover/274917/4 "2021-06-06T18:13:29Z")

</div>

Hmmm... I right understand when you execute query:  
GET develop1000/\_count  
result: 1000?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 6, 2021, 9:11pm UTC](https://discuss.elastic.co/t/number-of-documents-in-dev-tools-is-different-from-number-of-document-in-discover/274917/5 "2021-06-06T21:11:49Z")

</div>

> [@marone](#):
>
> I am expected to have 1000 documents in the index named `develop1000` , that's the result of `GET develop1000/_count` but the problem is when I check in Discover, I only have 992 hits.
> 
> What could be the problem please? and how to investigate the issue? I have the same problem in another index!
> 
> Thanks folks.

Perhaps some of the documents timestamps are not correct and inside / outside your time window in Discover OR somehow on the reindex the timestamp failed and those docs don't have a timestamp.

Also with 1000 documents just run the reindex in the foreground `wait_for_completion=true` and see if you get any errors

You can check what the min and max timestamp

```
GET develop1000/_search
{
  "size": 0,
  "aggs": {
    "min_date": {
      "min": {
        "field": "@timestamp",
        "format": "yyyy-MM-dd HH.mm.ss"
      }
    },
    "max_date": {
      "max": {
        "field": "@timestamp",
        "format": "yyyy-MM-dd HH.mm.ss"
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\
**Post date:** [June 7, 2021, 11:21pm UTC](https://discuss.elastic.co/t/number-of-documents-in-dev-tools-is-different-from-number-of-document-in-discover/274917/6 "2021-06-07T23:21:58Z")

</div>

Yes exactly!

---

<div class="post-metadata">

**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\
**Post date:** [June 7, 2021, 11:29pm UTC](https://discuss.elastic.co/t/number-of-documents-in-dev-tools-is-different-from-number-of-document-in-discover/274917/7 "2021-06-07T23:29:19Z")

</div>

Here is the result of reindexing with `wait_for_completion=true`:

```auto
{
  "took" : 8916,
  "timed_out" : false,
  "total" : 1000,
  "updated" : 0,
  "created" : 1000,
  "deleted" : 0,
  "batches" : 1,
  "version_conflicts" : 0,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : []
}

```

seems that there is no error. Then I execute it the search query to know the max and min timestamp:

```auto
{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1000,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : []
  },
  "aggregations" : {
    "max_date" : {
      "value" : 1.603826335E12,
      "value_as_string" : "2020-10-27 19.18.55"
    },
    "min_date" : {
      "value" : 1.603826285E12,
      "value_as_string" : "2020-10-27 19.18.05"
    }
  }
}

```

What you said is interesting, how can I check if a given document have empty timestamp?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 8, 2021, 12:24am UTC](https://discuss.elastic.co/t/number-of-documents-in-dev-tools-is-different-from-number-of-document-in-discover/274917/8 "2021-06-08T00:24:17Z")

</div>

> **[Exists query | Elasticsearch Guide \[7.13\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html)**

What is your index pattern time field?

`@timestamp` or the field `value` you show above. Discover is based on the time field defined in the index pattern

---

<div class="post-metadata">

**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\
**Post date:** [June 9, 2021, 12:31pm UTC](https://discuss.elastic.co/t/number-of-documents-in-dev-tools-is-different-from-number-of-document-in-discover/274917/9 "2021-06-09T12:31:09Z")

</div>

For the time field it's during October 2020 for all events, in discover I set the time to look for 2 years ago to be sure to get all events. But as you suggested the problem is that the field `@timestamp` is missing for 8 documents and checked that using the following query, as you pointed out in your answer:

```auto
GET develop1000/_search
{
  "query": {
    "bool": {
      "must_not": {
        "exists": {
          "field": "@timestamp"
        }
      }
    }
  }
}

```

the returned result is exactly 8 documents which they don't have a `@timestamp` field:

```auto
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 8,
      "relation" : "eq"
    },
    "max_score" : 0.0,
    "hits" : [
         // details for the 8 documents ...
      ]
}

```

Thank you @stephenb ! 😀

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 7, 2021, 12:31pm UTC](https://discuss.elastic.co/t/number-of-documents-in-dev-tools-is-different-from-number-of-document-in-discover/274917/10 "2021-07-07T12:31:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.