Duplicate documents with exactly same index/type/id in ES5.4.2

Elastic Stack Elasticsearch
1

My ES cluster has only one node. The "model" index has 5 primary shards and 1 replica shard.

After indexing several documents, I run the below query:

GET https://9.111.255.174:9200/model/cubes/_search

The response is as below.

{
  "took": 21,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 1,
    "hits": [
      {
        "_index": "model",
        "_type": "cubes",
        "_id": "slot_allocation_cube_1",
        "_score": 1,
        "_source": {
         ......
        }
      },
      {
        "_index": "model",
        "_type": "cubes",
        "_id": "slot_allocation_cube_1",
        "_score": 1,
        "_source": {
        ......
        }
      }
    ]
  }
}

We can see that there are two duplicated documents with the same "_id" as "slot_allocation_cube_1". I'm sure I didn't use customer routing during data indexing.

After investigation, I find the data is correct on shard 0. However, there is something wrong with shard 3.

I run the below query

GET https://9.111.255.174:9200/model/cubes/_search?preference=_shards: 3

The response is as below.

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "model",
        "_type": "cubes",
        "_id": "slot_allocation_cube_1",
        "_score": 1,
        "_source": {
         ......
        }
      }
    ]
  }
}

However, when I specify the id and run the below query

GET https://9.111.255.174:9200/model/cubes/slot_allocation_cube_1?preference=_shards:3

It failed!!!

 {
      "error": {
        "root_cause": [
          {
            "type": "remote_transport_exception",
            "reason": "[client_jfp4][9.111.255.174:9300][indices:data/read/get[s]]"
          }
        ],
        "type": "null_pointer_exception",
        "reason": null
      },
      "status": 500
    }

It is really strange. Can someone help me on this? Thanks a million!!!

2

Could anyone help on this issue?

:sob::sob::sob::sob::sob::sob::sob::sob::sob::sob::sob:

3

What do the logs show when you get that 500 error?

What does the search show if you add ?explain=true to the end? It should show which shard it is hitting and more.

Can you show your index settings as well?