Docs Count vs Index_total

[jonessmithville]

I am trying to find a way to determine how fast or if all documents are getting indexed timely. I checked the previous days index ---

GET indexname-2017.12.27/_stats

and noticed that

docs": {
"count": 19695896,

Is much higher than
"index_total": 11819794,
(no deletes)
It seems then there are ~8m docs that didn't get indexed. Anything I am missing anything here?

[luiz.santos]

Hi @jonessmithville,

The doc.count represents the number of documents indexed in your index while index_total stands for number of indexing operations performed during elasticsearch uptime.

So if you update a document it will count as an indexing operation but your doc.count won't increase. Please, look at this example where I indexed 2 documents and updated 2 documents:

POST likes/doc/_bulk
{"index":{"_id": 1}}
{"likes": 10, "user": "john", "message": "elastic"}
{"index":{"_id": 2}}
{"likes": 20, "user": "john", "message": "elastic"}
{"index":{"_id": 1}}
{"likes": 5, "user": "ryan", "message": "elastic"}
{"index":{"_id": 2}}
{"likes": 10, "user": "ryan", "message": "elastic"}

We will observedoc.count = 2 and index_total = 4:

GET likes/_stats?human

{
  "_shards": {
    "total": 10,
    "successful": 5,
    "failed": 0
  },
  "_all": {
    "primaries": {
      "docs": {
        "count": 2,
        "deleted": 2
      },
      "store": {
        "size": "9.3kb",
        "size_in_bytes": 9544
      },
      "indexing": {
        "index_total": 4,
      ...
}

Hope it helps.

Cheers,
LG