# Delete large amount of data using \_delete\_by\_query

**URL:** <https://discuss.elastic.co/t/delete-large-amount-of-data-using--delete-by-query/79075>\
**Category:** Elasticsearch\
**Created:** [March 17, 2017, 6:32pm UTC](https://discuss.elastic.co/t/delete-large-amount-of-data-using--delete-by-query/79075 "2017-03-17T18:32:17Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![platform-team](https://avatars.discourse-cdn.com/v4/letter/p/4bbf92/32.png) [@platform-team](https://discuss.elastic.co/u/platform-team)\
**Post date:** [March 17, 2017, 6:32pm UTC](https://discuss.elastic.co/t/delete-large-amount-of-data-using--delete-by-query/79075/1 "2017-03-17T18:32:17Z")

</div>

Hello,

I am trying to remove a large amount of data from elasticsearch using \_delete\_by\_query. I've tried many options to get this to complete, but typically can only get several hundred (out of millions) of records to actually delete. The most common error is:

```
{
  "took": 3307,
  "timed_out": false,
  "total": 140739907,
  "deleted": 102,
  "batches": 1,
  "version_conflicts": 77,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1,
  "throttled_until_millis": 0,
  "failures": [
    {
      "index": "logstash-2017.02.28",
      "type": "fluentd",
      "id": "AVqCEZwbJhYoxRiSZL1-",
      "cause": {
        "type": "es_rejected_execution_exception",
        "reason": "rejected execution of org.elasticsearch.transport.TransportService$7@1ac5f094 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@795ac56a[Running, pool size = 2, active threads = 2, queued tasks = 50, completed tasks = 464576]]"
      },
      "status": 429
    }

```

Here is an example of a query. I've tried many different settings to no avail.

Anyone know how to get this to just slowly crawl through and remove all matching records, versus erroring out?

```
GET logstash-*/_delete_by_query?conflicts=proceed
{
  "query": { 
    "query_string": {
      "default_field": "log",
      "analyze_wildcard": true, 
      "query": "DEV-*"
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 17, 2017, 7:23pm UTC](https://discuss.elastic.co/t/delete-large-amount-of-data-using--delete-by-query/79075/2 "2017-03-17T19:23:51Z")

</div>

Could you try to run the query per index instead of using a wildcard ?

FWIW if you end up removing a lot of docs it could be better to reindex the documents which will remain instead.

---

<div class="post-metadata">

**Author:** ![platform-team](https://avatars.discourse-cdn.com/v4/letter/p/4bbf92/32.png) [@platform-team](https://discuss.elastic.co/u/platform-team)\
**Post date:** [March 17, 2017, 7:35pm UTC](https://discuss.elastic.co/t/delete-large-amount-of-data-using--delete-by-query/79075/3 "2017-03-17T19:35:42Z")

</div>

Thanks @dadoonet -- I'll give that a try and report back 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 14, 2017, 7:35pm UTC](https://discuss.elastic.co/t/delete-large-amount-of-data-using--delete-by-query/79075/4 "2017-04-14T19:35:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.