# Problem with deleted documents

**URL:** <https://discuss.elastic.co/t/problem-with-deleted-documents/55041>\
**Category:** Elasticsearch\
**Created:** [July 8, 2016, 11:17am UTC](https://discuss.elastic.co/t/problem-with-deleted-documents/55041 "2016-07-08T11:17:06Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![eacoimbra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eacoimbra/32/3518_2.png) [@eacoimbra](https://discuss.elastic.co/u/eacoimbra)\
**Post date:** [July 8, 2016, 11:17am UTC](https://discuss.elastic.co/t/problem-with-deleted-documents/55041/1 "2016-07-08T11:17:06Z")

</div>

Sorry for the silly question, but it is possible to delete Elasticsearch documents forever?

I'm in the migration process 1.7 -\> 2.3 and when I run the migration plugin all my indices were listed with fields with dots. After analysis, I identified these fields with dots were generated by a plugin installed on my Graylog server and that has no more use for me.

Instead of bothering to rename these fields, I decided to go for something more practical and delete all documents with these fields.

So I ran:

```
    for i in `curl -s -XGET localhost:9200/_all/ | jq -r "keys[]"`; do
        echo -n "Excluindo JMX do índice $i: "
        curl -XDELETE localhost:9200/$i/_query?q=label:JMX
        echo
    done

```

then...

```
#curl 'localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open graylog2_2 3 1 19925062 75050 15.7gb 7.8gb
green open graylog2_5 3 1 20001004 6792 18.3gb 9.1gb
green open graylog2_21 3 1 19952265 48034 17.3gb 8.6gb
green open graylog2_34 3 1 19968203 32270 17.9gb 8.9gb
green open graylog2_26 3 1 19950723 50344 17.5gb 8.7gb
green open graylog2_6 3 1 19976215 24772 18.3gb 9.1gb
green open graylog2_29 3 1 19962504 37908 17.9gb 8.9gb
green open graylog2_27 3 1 19953287 47122 17.5gb 8.7gb
green open graylog2_10 3 1 19928390 71802 17.4gb 8.7gb
green open graylog2_25 3 1 19950258 51176 17.5gb 8.7gb
green open graylog2_0 3 1 19980897 19551 15gb 7.5gb
green open graylog2_33 3 1 19965352 35020 17.8gb 8.9gb
green open graylog2_35 3 1 18677635 40233 17.1gb 8.4gb
green open graylog2_9 3 1 19958295 42350 17.2gb 8.6gb
green open graylog2_24 3 1 19956467 44080 17.4gb 8.7gb
green open graylog2_18 3 1 19956737 45109 17.4gb 8.7gb
green open graylog2_12 3 1 19955303 44811 17.4gb 8.7gb
green open graylog2_8 3 1 19956070 44784 17.9gb 8.9gb
green open graylog2_15 3 1 19926718 74654 17.4gb 8.7gb
green open graylog2_17 3 1 19957205 42997 17.3gb 8.6gb
green open graylog2_4 3 1 19988129 13418 18gb 9gb
green open graylog2_7 3 1 19955716 45062 17.8gb 8.9gb
green open graylog2_11 3 1 19945574 56475 18gb 9gb
green open graylog2_22 3 1 19965180 38857 17.4gb 8.7gb
green open graylog2_31 3 1 19957684 43390 17.6gb 8.8gb
green open graylog2_28 3 1 19960169 41356 17.3gb 8.6gb
green open graylog2_16 3 1 19953562 46613 17.4gb 8.7gb
green open graylog2_13 3 1 19946695 54502 17.5gb 8.7gb
green open graylog2_32 3 1 19958744 41386 17.7gb 8.8gb
green open graylog2_19 3 1 19958161 43828 17.5gb 8.7gb
green open graylog2_1 3 1 4529877 4007 3.5gb 1.7gb
green open graylog2_20 3 1 19957822 43082 17.4gb 8.7gb
green open graylog2_3 3 1 4458723 7190 3.5gb 1.7gb
green open graylog2_14 3 1 19955648 45826 17.4gb 8.7gb
green open graylog2_23 3 1 19957960 43160 17.3gb 8.6gb
green open graylog2_30 3 1 19960653 40325 17.8gb 8.9gb

```

... and finally:

```
#curl -XPOST localhost:9200/_optimize?only_expunge_deletes=true

```

When I do a search on Graylog, I can not find these documents anymore. Great! But when I run the migration plugin again, it keeps warning that I own fields with dots and are the same fields.

Where did I go wrong?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 8, 2016, 11:37am UTC](https://discuss.elastic.co/t/problem-with-deleted-documents/55041/2 "2016-07-08T11:37:07Z")

</div>

The mapping is not updated (and you can't update it to remove a field).

That's why you have that.

You need to reindex in another index.  
If you are using rolling indices, then just remove old indices and don't create non fields in the new ones.

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [July 8, 2016, 11:49am UTC](https://discuss.elastic.co/t/problem-with-deleted-documents/55041/3 "2016-07-08T11:49:44Z")

</div>

Fields are forever, unfortunately. You have to move to a new index that  
doesn't have them.

---

<div class="post-metadata">

**Author:** ![eacoimbra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eacoimbra/32/3518_2.png) [@eacoimbra](https://discuss.elastic.co/u/eacoimbra)\
**Post date:** [July 8, 2016, 12:08pm UTC](https://discuss.elastic.co/t/problem-with-deleted-documents/55041/4 "2016-07-08T12:08:48Z")

</div>

Thanks guys, I got it!

But the problem is... I did not create any of these indices, all of them were created by the application, such as Kibana does.  
I have no idea how to do this. Fear of losing data...  
😞

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:37pm UTC](https://discuss.elastic.co/t/problem-with-deleted-documents/55041/5 "2017-07-05T22:37:07Z")

</div>