# Deleted docs could be still retrieved although refreshed

**URL:** <https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609>\
**Category:** Elasticsearch\
**Created:** [November 23, 2022, 3:27am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609 "2022-11-23T03:27:33Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![fanqiaoqing](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fanqiaoqing/32/49698_2.png) [@fanqiaoqing](https://discuss.elastic.co/u/fanqiaoqing)\
**Post date:** [November 23, 2022, 3:27am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/1 "2022-11-23T03:27:33Z")

</div>

Hi Elasticsearch,

I have an index with about 2.5 billion documents. The primary has a total of 10 shards, and each shard is about 10G.

We have a problem like this:  
A query will find out the number of documents for some query conditions. We tried to delete one document, and then use the same query to update the result, and find that the document we just deleted is still exist in the hits collection.

Actually, the refresh interval setting of this index is 1s, we waited for 1min, and run the query, the result is still not updated.

We are a little confused, is there any documentation that can help us understand this behavior of Elasticsearch, that would be very appreciated.

Thanks  
Qiaoqing.

---

<div class="post-metadata">

**Author:** ![fanqiaoqing](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fanqiaoqing/32/49698_2.png) [@fanqiaoqing](https://discuss.elastic.co/u/fanqiaoqing)\
**Post date:** [November 23, 2022, 3:38am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/2 "2022-11-23T03:38:21Z")

</div>

Provide some desensitizing context to that could help you understand,

```auto
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open myindex qgzM6RmHSdCTgSsacSBauw 10 2 2408738470 0 214.8gb 107.6gb

```

index settings

```auto
{
  "myindex" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            },
            "total_shards_per_node" : "2"
          }
        },
        "refresh_interval" : "1s",
        "number_of_shards" : "10",
        "provided_name" : "myindex",
        "creation_date" : "1669104181455",
        "unassigned" : {
          "node_left" : {
            "delayed_timeout" : "15m"
          }
        },
        "number_of_replicas" : "2",
        "uuid" : "qgzM6RmHSdCTgSsacSBauw",
        "version" : {
          "created" : "7100299"
        }
      }
    }
  }
}

```

delete query

```auto
DELETE /myindex/_doc/the_doc_id

```

aggregation query

```auto
POST /myindex/_search?routing=myroutingkey&typed_keys=true&max_concurrent_shard_requests=5&search_type=query_then_fetch&batched_reduce_size=512
{
	"from": 0,
	"size": 100,
	"query": {
		"bool": {
			"filter": [
				{
					"term": {
						"some_field": {
							"value": "somevalue",
							"boost": 1.0
						}
					}
				},
				{
					"term": {
						"some_field": {
							"value": "somevalue",
							"boost": 1.0
						}
					}
				},
				{
					"nested": {
						"query": {
							"bool": {
								"filter": [
									{
										"term": {
											"some_field": {
												"value": "somevalue",
												"boost": 1.0
											}
										}
									}
								],
								"adjust_pure_negative": true,
								"boost": 1.0
							}
						},
						"path": "properties",
						"ignore_unmapped": false,
						"score_mode": "none",
						"boost": 1.0
					}
				}
			],
			"must_not": [
				{
					"term": {
						"some_field": {
							"value": "somevalue",
							"boost": 1.0
						}
					}
				}
			],
			"adjust_pure_negative": true,
			"boost": 1.0
		}
	},
	"version": true,
	"explain": false,
	"sort": [
		{
			"some_field": {
				"order": "desc"
			}
		}
	],
	"aggregations": {
		"aggs_name1": {
			"nested": {
				"path": "some_field"
			},
			"aggregations": {
				"agg_name2": {
					"filter": {
						"range": {
							"some_field": {
								"from": 123456,
								"to": null,
								"include_lower": false,
								"include_upper": true,
								"boost": 1.0
							}
						}
					},
					"aggregations": {
						"agg_name3": {
							"terms": {
								"field": "some_field",
								"size": 100,
								"min_doc_count": 1,
								"shard_min_doc_count": 0,
								"show_term_doc_count_error": false,
								"order": {
									"_key": "desc"
								}
							},
							"aggregations": {
								"agg_name4": {
									"sum": {
										"field": "some_field"
									}
								},
								"performanceDateLifetimeGainOrLoss": {
									"sum": {
										"field": "some_field"
									}
								}
							}
						}
					}
				}
			}
		},
		"agg_name5": {
			"sum": {
				"field": "some_field"
			}
		},
		"agg_name6": {
			"sum": {
				"field": "some_field"
			}
		}
	}
}

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 23, 2022, 4:24am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/3 "2022-11-23T04:24:17Z")

</div>

> [@fanqiaoqing](#):
>
> ```auto
> DELETE /myindex/_doc/the_doc_id
> 
> ```

And when you delete it and wait a minute (or the refresh time), can you `GET /myindex/_doc/the_doc_id` and it provides that document?

---

<div class="post-metadata">

**Author:** ![fanqiaoqing](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fanqiaoqing/32/49698_2.png) [@fanqiaoqing](https://discuss.elastic.co/u/fanqiaoqing)\
**Post date:** [November 23, 2022, 4:45am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/4 "2022-11-23T04:45:01Z")

</div>

before executing that query, I delete the doc and then it still shows in the hits result.

If I try this command,

```auto
GET /myindex/_doc/the_doc_id

```

It will return 404.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 23, 2022, 5:06am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/5 "2022-11-23T05:06:36Z")

</div>

Which version of Elasticsearch are you using?

How can you tell from the aggregation that the document is still there?

---

<div class="post-metadata">

**Author:** ![fanqiaoqing](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fanqiaoqing/32/49698_2.png) [@fanqiaoqing](https://discuss.elastic.co/u/fanqiaoqing)\
**Post date:** [November 23, 2022, 5:36am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/6 "2022-11-23T05:36:01Z")

</div>

7.10

I tried to delete the doc using the DELETE API

```auto
DELETE /myindex/_doc/the_doc_id

```

and then use the above query to get the latest result, the same doc with the doc id **the\_doc\_id** is showed in the **hits** part, although GET /myindex/\_doc/the\_doc\_id returns 404

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 23, 2022, 5:52am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/7 "2022-11-23T05:52:58Z")

</div>

Are you using (or have used) routing on this index? Can you try using a few different [preference strings](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/search-shard-routing.html#shard-and-node-preference) in your command, e.g. `GET /myindex/_doc/the_doc_id?preference=preferencestring00001` and see if you find it on any other shard?

Maybe you could try a simple [id query](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/query-dsl-ids-query.html) across all shards?

---

<div class="post-metadata">

**Author:** ![fanqiaoqing](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fanqiaoqing/32/49698_2.png) [@fanqiaoqing](https://discuss.elastic.co/u/fanqiaoqing)\
**Post date:** [November 23, 2022, 6:01am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/8 "2022-11-23T06:01:38Z")

</div>

Thank you very much, Christian!

Yes.  
I tried the three queries.

GET /myindex/\_doc/the\_doc\_id?preference=preferencestring00001 returns 404  
GET /myindex/\_doc/the\_doc\_id returns 404

but the id query

```auto
GET /_search
{
  "query": {
    "ids" : {
      "values" : ["1", "4", "100"]
    }
  }
}

```

returns 200.

We are a little confused why the refresh interval is not working, could you please share with us some insights, or any documentations we can check.  
We need to figure out some ways to avoid this.

THanks,  
Qiaoqing.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 23, 2022, 6:04am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/9 "2022-11-23T06:04:10Z")

</div>

Can you share the metadata of the documents you found when using the ID query?

---

<div class="post-metadata">

**Author:** ![fanqiaoqing](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fanqiaoqing/32/49698_2.png) [@fanqiaoqing](https://discuss.elastic.co/u/fanqiaoqing)\
**Post date:** [November 23, 2022, 6:08am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/10 "2022-11-23T06:08:25Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> Can you share the metadata of the documents you found when using the ID query?

Metadata?  
you mean the response body?

it was like this

```auto
{
    "took" : 1133,
    "timed_out" : false,
    "_shards" : {
      "total" : 84,
      "successful" : 84,
      "skipped" : 0,
      "failed" : 0
    },
    "hits" : {
      "total" : {
        "value" : 1,
        "relation" : "eq"
      },
      "max_score" : 1.0,
      "hits" : [
        {
          "_index" : "myindex",
          "_type" : "_doc",
          "_id" : "the_doc_id",
          "_score" : 1.0,
          "_routing" : "the_routing_key",
          "_source" : {}
        }
      ]
    }
  }
  

```

And the

```auto
 GET /myindex/_doc/the_doc_id returns
{
  "_index" : "myindex",
  "_type" : "_doc",
  "_id" : "the_doc_id",
  "found" : false
}

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 23, 2022, 6:24am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/11 "2022-11-23T06:24:09Z")

</div>

What does `GET /myindex/_doc/the_doc_id?routing=the_routing_key` return (replace document ID and routing key with real values)?

If you index a document using routing, you need to use that routing key for all operations on that document, e.g. updates or deletes, as this determines which shard the document gets written to. This has nothing to do with the refresh interval.

If you index a document with routing key `abc` it may get written to shard 1 even though the ID would have had it indexed into shard 5. If you just delete the document based on ID this request will go to shard 5 if you do not specify the same routing key and the document will not be deleted.

---

<div class="post-metadata">

**Author:** ![fanqiaoqing](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fanqiaoqing/32/49698_2.png) [@fanqiaoqing](https://discuss.elastic.co/u/fanqiaoqing)\
**Post date:** [November 23, 2022, 6:29am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/12 "2022-11-23T06:29:31Z")

</div>

Thanks Christian.

`GET /myindex/_doc/the_doc_id?routing=the_routing_key` will return 200.

So should I also use routing key when I deleted the doc?

```auto
DELETE /myindex/_doc/the_doc_id?routing=the_routing_key

```

Or in other words,  
How can I delete the same doc on all shards if I indexed one doc with routing key.

Thanks,  
Qiaoqing.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 23, 2022, 6:33am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/13 "2022-11-23T06:33:37Z")

</div>

You need to use the correct routing key when deleting the document like in your example. The only way to delete from all shards, without the routing key, is to use delete by query with an ID clause, but note that this is a lot more expensive than a direct delete using the routing key. This is an important aspect to consider when you adopt routing.

---

<div class="post-metadata">

**Author:** ![fanqiaoqing](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fanqiaoqing/32/49698_2.png) [@fanqiaoqing](https://discuss.elastic.co/u/fanqiaoqing)\
**Post date:** [November 23, 2022, 6:34am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/14 "2022-11-23T06:34:32Z")

</div>

Thanks Christian and appreciate your patience.  
I see, we will apply the changes in our code.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 23, 2022, 6:44am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/15 "2022-11-23T06:44:52Z")

</div>

As you are running an old version that is EOL I would also recommend you upgrade to at least 7.17.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 21, 2022, 6:45am UTC](https://discuss.elastic.co/t/deleted-docs-could-be-still-retrieved-although-refreshed/319609/16 "2022-12-21T06:45:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.