Number of documents in a Kibana search does not match Elasticsearch

Elastic Stack Kibana
1

I'm using an ELK stack, all components are version 7.10

I'm using /_cat/indices/logset* to query elasticsearch on how many docs I have in my index. This is the only index I have currently as below:

health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   logset1 g_4st7jaT6m8RkOprTT1Rw   6   1    75057890            0      1.5gb        796.4mb

As you see I have 75057890 docs according to elasticsearch

When I go to Kibana, without entering any search parameters, it says I have 71191355 docs. So I'm missing around 4 million docs.

image
image844×163 7.46 KB

The date range selected (1 year) easily covers the log data I have. I need to find out the reason for this discrepancy. How do I begin debugging this?

2

Are you using nested docs in your mapping?

Ideally, if you want to compare the number, use:

GET /logset1/_count

BTW, please upgrade your version ASAP. At least to 7.17.

3

No I am not using any nested mappings. The count api yields exactly the same result

{
  "count": 75057890,
  "_shards": {
    "total": 6,
    "successful": 6,
    "skipped": 0,
    "failed": 0
  }
}

Also, why the urgency to upgrade?

4

The count api yields exactly the same result

Ok. So there's probably a filter or a time range which is sent by Kibana.
There's somewhere an inspect button which helps to see what exactly is the query ran by Kibana.

Also, why the urgency to upgrade?

Security patches as the first goal. But also much more stability, bug fixes over the last 3+ years...

5

You can also run this which Will show them min and Max time frame.

POST /logset1/_search
{
  "size": 0,
  "aggs": {
    "count": {
      "value_count": {
        "field": "@timestamp"
      }
    },
    "min_timestamp": {
      "min": {
        "field": "@timestamp"
      }
    },
    "max_timestamp": {
      "max": {
        "field": "@timestamp"
      }
    }
  }
}

It's also possible that you have logs that do not have timestamp and thus won't show up in Kibana Discover

Or documents with malformed timestamp that are outside your search

6

Thanks! That gave me a really big clue. Turns out many records did not parse correctly and resulted in grokparsefailure.

As a result there was no timestamp field on these records, which explains the discrepancy.

1 Like