Incorrect Doc Count vs index total returns in ES

[Wayne_Taylor]

Hi team,

Using ES 7.1.1 and noticing a weird count of documents shown in Kibana vs the reporting totals from ES APIs and Elastic HQ.

Specifically, I have an index pattern created daily for Jaeger spans. This is showing over 700 million entries (which is crazy). However, if I use Kibana I am more like 20 Million.

As i understand from : Docs Count vs Index_total. That doc count is total documents where as total_index = number of operations including deletes and updates.

Here is difference from raw ES Index stat API and ES count API. Notice ES count API seems more in line with whats expected

Is this a known bug or did i miss something?

Wayne

[dadoonet (David Pilato)]

Are you using nested data type in your mapping by any chance?

[Wayne_Taylor]

Hi @dadoonet - yes. Jaeger uses a lot of nested data types for key values pairs e.g. process has lots of sub items such as process.servicename and even sub nests in there.

Assuming based upon your response this count ?

Wayne

[dadoonet (David Pilato)]

Every nested document is a Lucene document indeed. But from elasticsearch, only the "parent" document is reported when you search.

[Wayne_Taylor]

Thanks @dadoonet. This makes a lot more sense as to why our spans index reported this way then. Good to know. Marking as the resolution and appreciate your response

Wayne