Incorrect/increasing elasticsearch count query after re-indexing!

Elastic Stack Elasticsearch
1

Some hours ago I've re-indexed one of my indices. The re-indexing process completed in about 2 hours and I've run the following query afterwards:

GET myIndex/_count
{
  "query": {
    "bool": {
      "must": [
        {
          "exists": {
            "field": "productDefinitionKey.confidence"
          }
        }
      ]
    }
  }
}

The problem is that the result of this count query was about 6,000 but it must be about 2,000,000! I've checked multiple documents and seen all have the productDefinitionKey.confidence field. Now, after about 20 hours, the result of this query is about 100,000 and is increasing. I guess something is running in background but I don't know what! I've searched a lot and asked from my Elasticsearch geek friends but couldn't know what caused the problem and what is happening.

One of my colleagues said something about checking ingest nodes and proposed to check system resources. I did dot that but I guess resources are not the problem.
The below screenshots captured from elasticsearch vue, htop and nmon:

image
image1838×507 16.2 KB

image
image1222×218 104 KB

image
image1012×816 74 KB

2

Right now we understand the problem. The field we searched for it, i.e. productDefinitionKey.confidence was not in index template but after re-indexing one of our remote colleagues changed the index mapping and added mapping for this new field! I guess he did do that immediately after re-indexing was completed and run his code to add new field into existing docs.

So the count number we saw was the count of docs that his code updated and all of added fields by re-indexing gone just to _source and hadn't been indexed! Because we set dynamic to false in the template beforehand.

2 Likes
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.