# Delete by query deletes only 1000 documents, then quits

**URL:** <https://discuss.elastic.co/t/delete-by-query-deletes-only-1000-documents-then-quits/350529>\
**Category:** Elasticsearch\
**Created:** [January 7, 2024, 11:15am UTC](https://discuss.elastic.co/t/delete-by-query-deletes-only-1000-documents-then-quits/350529 "2024-01-07T11:15:00Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![d8d4a522fb1d394d9705](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d8d4a522fb1d394d9705/32/123759_2.png) [@d8d4a522fb1d394d9705](https://discuss.elastic.co/u/d8d4a522fb1d394d9705)\
**Post date:** [January 7, 2024, 11:15am UTC](https://discuss.elastic.co/t/delete-by-query-deletes-only-1000-documents-then-quits/350529/1 "2024-01-07T11:15:00Z")

</div>

I am using the following the api to delete documents older than 60 days:

```auto
POST /index_name/_delete_by_query?conflicts=proceed
{
   "query": {
     "range": { 
      "@timestamp": {"lte": "now-60d/d"}
    }
  }
}

```

My index is quite big in size: 1.2tb. When I run this api, it deletes max 1000 documents and then quits.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [January 7, 2024, 11:51am UTC](https://discuss.elastic.co/t/delete-by-query-deletes-only-1000-documents-then-quits/350529/2 "2024-01-07T11:51:35Z")

</div>

Welcome.

That's really inefficient. A delete request actually writes more data on disk and eventually removes it.

Instead use time based indices and simply delete the indices you don't need anymore.  
You can use ILM to automate all that.

See [ILM: Manage the index lifecycle | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html)

---

<div class="post-metadata">

**Author:** ![d8d4a522fb1d394d9705](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d8d4a522fb1d394d9705/32/123759_2.png) [@d8d4a522fb1d394d9705](https://discuss.elastic.co/u/d8d4a522fb1d394d9705)\
**Post date:** [January 7, 2024, 12:17pm UTC](https://discuss.elastic.co/t/delete-by-query-deletes-only-1000-documents-then-quits/350529/3 "2024-01-07T12:17:17Z")

</div>

I know index deletion is faster. In fact we are planning to have daily indices in near future. However till then I have to remove old documents. Will you please help me optimize query if there is any way?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [January 7, 2024, 12:38pm UTC](https://discuss.elastic.co/t/delete-by-query-deletes-only-1000-documents-then-quits/350529/4 "2024-01-07T12:38:29Z")

</div>

I think it'd be better to reindex the data you want to keep instead.

What is your version?

---

<div class="post-metadata">

**Author:** ![d8d4a522fb1d394d9705](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d8d4a522fb1d394d9705/32/123759_2.png) [@d8d4a522fb1d394d9705](https://discuss.elastic.co/u/d8d4a522fb1d394d9705)\
**Post date:** [January 7, 2024, 12:52pm UTC](https://discuss.elastic.co/t/delete-by-query-deletes-only-1000-documents-then-quits/350529/5 "2024-01-07T12:52:05Z")

</div>

Version-7.17.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [January 7, 2024, 1:16pm UTC](https://discuss.elastic.co/t/delete-by-query-deletes-only-1000-documents-then-quits/350529/6 "2024-01-07T13:16:18Z")

</div>

You can try adding

```
wait_for_completion=false

```

So it will run asynchronously.

---

<div class="post-metadata">

**Author:** ![d8d4a522fb1d394d9705](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d8d4a522fb1d394d9705/32/123759_2.png) [@d8d4a522fb1d394d9705](https://discuss.elastic.co/u/d8d4a522fb1d394d9705)\
**Post date:** [January 7, 2024, 1:47pm UTC](https://discuss.elastic.co/t/delete-by-query-deletes-only-1000-documents-then-quits/350529/7 "2024-01-07T13:47:21Z")

</div>

I tried that and I got below result:

```auto
{
  "completed" : true,
  "task" : {
    "node" : "Z19SnYRVRTqf9G_kgaC4Yg",
    "id" : 1623007937,
    "type" : "transport",
    "action" : "indices:data/write/delete/byquery",
    "status" : {
      "total" : 18231930,
      "updated" : 0,
      "created" : 0,
      "deleted" : 1000,
      "batches" : 1,
      "version_conflicts" : 0,
      "noops" : 0,
      "retries" : {
        "bulk" : 0,
        "search" : 0
      },
      "throttled_millis" : 0,
      "requests_per_second" : -1.0,
      "throttled_until_millis" : 0
    },
    "description" : "delete-by-query [index-name]",
    "start_time_in_millis" : 1704633497441,
    "running_time_in_nanos" : 493323541261,
    "cancellable" : true,
    "cancelled" : false,
    "headers" : { }
  },
  "error" : {
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : -1,
        "index" : null,
        "reason" : {
          "type" : "search_context_missing_exception",
          "reason" : "No search context found for id [30575029]"
        }
      }
    ],
    "caused_by" : {
      "type" : "search_context_missing_exception",
      "reason" : "No search context found for id [30575029]"
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [January 8, 2024, 9:14am UTC](https://discuss.elastic.co/t/delete-by-query-deletes-only-1000-documents-then-quits/350529/8 "2024-01-08T09:14:41Z")

</div>

Not sure what is happening on your cluster. Is it overloaded at the moment?

What is the output of:

```auto
GET /
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v

```

If some outputs are too big, please share them on [gist.github.com](http://gist.github.com) and link them here.

May be you could reduce the `scroll_size` to 100 and try again? Or increase `scroll` to `1m` (I don't remember what the default value is).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 5, 2024, 9:14am UTC](https://discuss.elastic.co/t/delete-by-query-deletes-only-1000-documents-then-quits/350529/9 "2024-02-05T09:14:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.