# Removing unneeded documents

**URL:** <https://discuss.elastic.co/t/removing-unneeded-documents/172825>\
**Category:** Elasticsearch\
**Created:** [March 18, 2019, 4:21pm UTC](https://discuss.elastic.co/t/removing-unneeded-documents/172825 "2019-03-18T16:21:21Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jkaoiadjmdna](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@jkaoiadjmdna](https://discuss.elastic.co/u/jkaoiadjmdna)\
**Post date:** [March 18, 2019, 4:21pm UTC](https://discuss.elastic.co/t/removing-unneeded-documents/172825/1 "2019-03-18T16:21:22Z")

</div>

We manage a small cluster that has been collecting logs from Kubernetes for a few months which includes typical application JSON logs, but also kubernetes health metrics. These make up the vast majority of our records by quantity, and I'd be willing to bet by on-disk space as well.

I would like to remove _all_ of these unwanted logs but leave the others untouched. I was experimenting with something like

```auto
POST /_all/_delete_by_query
{
  "query" : {
		"match": {
			"kubernetes.namespace": {
				"query": "kube-system"
			}
		}
  }
}

```

But this might not do what I expect (it certainly seems limited to 1000 documents at a time which won't suffice for my 1bln+ records). While we research how to best never add these logs in the first place, what is the right strategy for removing the ones that match the above query?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 18, 2019, 4:57pm UTC](https://discuss.elastic.co/t/removing-unneeded-documents/172825/2 "2019-03-18T16:57:11Z")

</div>

Using delete by query API to delete 1bln+ documents is definitely a costly operation.  
Worth considering reindexing the other documents in a new index may be instead?

---

<div class="post-metadata">

**Author:** ![jkaoiadjmdna](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@jkaoiadjmdna](https://discuss.elastic.co/u/jkaoiadjmdna)\
**Post date:** [March 18, 2019, 6:22pm UTC](https://discuss.elastic.co/t/removing-unneeded-documents/172825/3 "2019-03-18T18:22:30Z")

</div>

I'm not at all experienced with ES, so that's possible. I'm not well versed in this system, and what I need to do to reduce the consumed space of uneeded logs that are intermixed in an index with ones I do want. Can you point me to an example or documentation that is relevant to my example? Even some pseudo-code of the actions I need to take so I have the proper terminology when learning more about it myself would be much appreciated!

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 18, 2019, 7:30pm UTC](https://discuss.elastic.co/t/removing-unneeded-documents/172825/4 "2019-03-18T19:30:25Z")

</div>

I was thinking of something like this:

```auto
POST _reindex
{
  "source": {
    "index": "*"
  },
  "dest": {
    "index": "tmp_index_name"
  },
  "query": {
    "bool": {
      "must_not": [
        {
          "match": {
            "kubernetes.namespace": {
              "query": "kube-system"
            }
          }
        }
      ]
    }
  },
  "script": {
    "source": "ctx._index=\"new_\"+ctx._index",
    "lang": "painless"
  }
}

```

If you are using daily indices with a short time of retention, you can also just wait for the oldest indices to be removed, may be?

---

<div class="post-metadata">

**Author:** ![jkaoiadjmdna](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@jkaoiadjmdna](https://discuss.elastic.co/u/jkaoiadjmdna)\
**Post date:** [April 10, 2019, 4:15pm UTC](https://discuss.elastic.co/t/removing-unneeded-documents/172825/5 "2019-04-10T16:15:18Z")

</div>

For anyone else, I essentially did this. My query was a bit more involved since I had a couple different namespaces. I used the python library to make it easier but this did the trick.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 8, 2019, 4:15pm UTC](https://discuss.elastic.co/t/removing-unneeded-documents/172825/6 "2019-05-08T16:15:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.