# How to partially delete an index

**URL:** <https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317>\
**Category:** Elasticsearch\
**Created:** [April 6, 2021, 10:03am UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317 "2021-04-06T10:03:17Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\
**Post date:** [April 6, 2021, 10:03am UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/1 "2021-04-06T10:03:18Z")

</div>

How can I delete only the records that meet the criteria of the index registered in elasticsearch?

I don't want to delete the entire index.  
In other words, not the following instruction.

```auto
curl -XDELETE localhost:9200/index_name?pretty=true

```

---

<div class="post-metadata">

**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\
**Post date:** [April 6, 2021, 10:06am UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/2 "2021-04-06T10:06:43Z")

</div>

In addition, please let me know if there is an option to run a test run (like dry-run, which doesn't actually delete the file, but checks the execution).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 6, 2021, 10:14am UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/3 "2021-04-06T10:14:30Z")

</div>

Have a look at:

> **[Delete by query API | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html)**

> [@its-ogawa](#):
>
> In addition, please let me know if there is an option to run a test run (like dry-run, which doesn't actually delete the file, but checks the execution).

Do a `_search` instead with the same exact query.

---

<div class="post-metadata">

**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\
**Post date:** [April 6, 2021, 10:21am UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/4 "2021-04-06T10:21:32Z")

</div>

Thank you for answering my question.

Does it support the curl command?  
I seem to get an error with the following command.

```auto
# curl -XPOST 'localhost:9200/api-2021.04.06/_search' -d '{ "query": { "match": { "log.file.path":"/var/log/api/api-2021-02" } } }'
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}

```

---

<div class="post-metadata">

**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\
**Post date:** [April 6, 2021, 11:19am UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/5 "2021-04-06T11:19:59Z")

</div>

First of all, I decided to do as you said and send a GET request from Kibana's developer tool.

The following request works.

```auto
GET /api-2021.04.06/_search
{
  "query": {
    "bool": {
      "must": {
        "match": {
          "log.file.path":"/var/log/api/api-2021-02-*.log"
        }
      }
    }
  }
}

```

However, the all-important delete request does not work.

```auto
POST /api-2021.04.06/_delete_by_query
{
  "query": {
    "bool": {
      "must": {
        "match": {
          "log.file.path":"/var/log/api/api-2021-02-*.log"
        }
      }
    }
  }
}

```

```auto
{"statusCode":502,"error":"Bad Gateway","message":"Client request timeout"}

```

It means a timeout, but I don't believe it is that big a log. What could be the cause? Also, is there any way to extend the timeout period?

---

<div class="post-metadata">

**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\
**Post date:** [April 6, 2021, 11:40am UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/6 "2021-04-06T11:40:22Z")

</div>

If I run it from elasticsearch-head, I get another error.

```auto
"failures": [
{
... snip ...
"cause": {
"type": "cluster_block_exception",
"reason": "index [api-2021.04.06] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"
},
"status": 429
}

```

I saw this same error in Kibana when I had a lot of logs fetched by filebeat and the disk was getting tight.

However, I am trying to delete the log due to this error confirmed by Kibana.  
If this operation is also blocked, how can I reduce the disk space?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 6, 2021, 11:48am UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/7 "2021-04-06T11:48:50Z")

</div>

DELETE a document does not remove the doc immediately. It creates more data on disk to Mark the document as deleted.

Then it eventually removes it when a merge happens.

Because you did not tell initially, may be describe what is the pro lyon want to fix by deleting some documents?

---

<div class="post-metadata">

**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\
**Post date:** [April 6, 2021, 11:58am UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/8 "2021-04-06T11:58:45Z")

</div>

I am sorry that I did not explain it well enough.

> Because you did not tell initially, may be describe what is the pro lyon want to fix by deleting some documents?

What do you mean by this?

Does it mean that it is not practical to request a delete query from elasticsearch in this situation?

Please let me know if you have any good ideas to solve [TOO\_MANY\_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block].

---

<div class="post-metadata">

**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\
**Post date:** [April 6, 2021, 12:21pm UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/9 "2021-04-06T12:21:40Z")

</div>

> Then it eventually removes it when a merge happens.  
> Perhaps you may have found yourself in the situation described here.

The following error has occurred.

```auto
"failures": [
{
... snip ...
"cause": {
"type": "version_conflict_engine_exception",
"reason": "[3NYwpngBPtRZ_HR4DA5t]: version conflict, required seqNo [11709405], primary term [1]. but no document was found",
"index_uuid": "4FRCHYUyTzWKrYs1fmr2Aw",
"shard": "0",
"index": "api-2021.04.06"
},
"status": 409
}

```

Is there any way to resolve the conflict?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 6, 2021, 3:56pm UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/10 "2021-04-06T15:56:42Z")

</div>

What is the current status of your cluster?

What is the output of:

```auto
GET /
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v

```

If some outputs are too big, please share them on [gist.github.com](http://gist.github.com) and link them here.

---

<div class="post-metadata">

**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\
**Post date:** [April 7, 2021, 1:31am UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/11 "2021-04-07T01:31:17Z")

</div>

Thank you for your answer.

I'm very sorry, but I couldn't wait for your answer, so I deleted the corresponding index, and now I can't reproduce it.

I think all the current statuses are in a good state.  
I will share the current status.

> <https://gist.github.com/its-ogawa/160825512afd43fe842cb8ac4c902431>

If the status is not healthy, what does this command tell you?

---

<div class="post-metadata">

**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\
**Post date:** [April 7, 2021, 7:17am UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/12 "2021-04-07T07:17:44Z")

</div>

I have deleted the index and can no longer check it. Is the following explanation correct?

```auto
GET /api-2021.04.06/_search
{
  "query": {
    "bool": {
      "must": {
        "match": {
          "log.file.path":"/var/log/api/api-2021-02-*.log"
        }
      }
    }
  }
}

```

However, the all-important delete request does not work.

```auto
POST /api-2021.04.06/_delete_by_query
{
  "query": {
    "bool": {
      "must": {
        "match": {
          "log.file.path":"/var/log/api/api-2021-02-*.log"
        }
      }
    }
  }
}

```

I would like to know this because in the future I may do something like partially deleting a document in the index.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 7, 2021, 9:12am UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/13 "2021-04-07T09:12:56Z")

</div>

> [@its-ogawa](#):
>
> what does this command tell you?

It just gives me an overview of your cluster to have a better understanding before trying to dig in things.

> [@its-ogawa](#):
>
> However, the all-important delete request does not work.

Why?

If you are running out of disk space, I can understand that but otherwise I don't see why it would not work.

---

<div class="post-metadata">

**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\
**Post date:** [April 9, 2021, 2:16am UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/14 "2021-04-09T02:16:11Z")

</div>

> If you are running out of disk space, I can understand that but otherwise I don't see why it would not work.

I'm sorry for the confusion.  
Your advice turned out to be correct.

```auto
[TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block].

```

The cause seems to be that elasticsearch is blocking index due to exhausted disk space.

We have secured enough disk space and changed the destination of the index in elasticsearch.  
After collecting the logs again and confirming that there were no errors, I ran the above command and it worked.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 7, 2021, 2:16am UTC](https://discuss.elastic.co/t/how-to-partially-delete-an-index/269317/15 "2021-05-07T02:16:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.